Twitch Extensions Part 5 – Dev Environment

This week we are actually going to write some code! Amazing I know! We’ll be using nodeJS and some basic shell scripting, here just for some simplicity. 

First off apologies for being 3 days late on this entry in the series!

In Part 4 we wrote about the Twitch Developer Rig and what it can/can’t do. One of the useful thing’s it can do for you is “host your files” for you when your extension is in Local test.

The Hosting options in the Developer rig.
The Hosting options in the Developer rig.

The Developer Rig, will either just “dumb serve” a folder of static files, or you can give it a full command to run, handy for WebPack/React/JS things that people need to pre-compile first.

But this big thing it won’t do is SSL Termination so whilst you can easily test your extension in the Rig, you won’t be able to easily test it on Twitch, which is the purpose of this little Dev Environment.

Personally at the moment I tend to write my Extensions in pure/vanilla JavaScript without libraries, since in most cases I’m just running a few fetch requests and drawing DOM elements, but the more interesting parts come with my compilation/bundling for hosting. The “rig” that I use is Developer Rig compatible since it is just a node command. But I’ve normally started it in a terminal as I’m testing on the actual Twitch website.

So what is the aim here?

To create a nodeJS Server that will

  • “static” host the HTML, JS and CSS for an extension,
  • do some clean up on JS/CSS, both for development and production,
  • work behind (a real) SSL for testing on the Twitch website (or rig)
  • be representative of Hosted test and above

What does that look like?

Well first we need to setup a bunch of folders, and we’ll set it up in a “nice” way for using Version control, some people may prefer to keep a separate repo for their EBS from their frontend for easier deploy. The choice is yours there! I use a mix, because being inconsistent is fun!

Proposed folder structure for your extension repository
Proposed folder structure for your extension repository

assets – for storing your screenshots, discovery images, icons and other bits and pieces that live on “Version Details”

ebs – the folder for building you EBS in

website – the folder for building a website in if your Extension has/needs one, usually would include your Privacy Policy.

extension – this is where our extension actually lives and is the folder we’ll be poking about in today.

The Extension Folder

The Folders in the Extension Folder
The Folders in the Extension Folder

assets – another assets folders? For storing any front end specific bits and pieces. You probably don’t need this.

build/release – build is where our “compiled” extension will sit

releases – I like to store my old/previous versions of the extension here for future reference

develop – the place we actually write our code

For Version Control, you would generally, touch build and release with a blank file (or .gitkeep if using Git) and then ignore those folders from version control.

We are going to be using the “static” part of NodeJS Express to serve the build folder, and use a super exciting bash script to populate the build folder from the develop folder.

Usually I’ll keep a dev folder in the develop folder, as I’ll keep the “pre-release” version of the extension in develop and the compiled/zip’ed version in releases.

The Bash Script though?

yeah, I use a bash script, it’s my preferred method, but anything it can do you can achieve in similar stuff such as WebPack, but you may want to run all sorts of things when you “deploy” you Extension Frontend during testing. And whilst I am considering other methods, I prefer the simple Bash script.

The Server

The server itself is relatively straight forward, you can refer to the Code on Github, but here is the key part we are interested in

const listen = 8050;
 
const express = require('express');
const app = express();
 
/*
Setup Express to Listen on a Port
*/
app.listen(listen, function () {
console.log('booted express on', listen);
})
 
/*
Setup a "Log" Event for file loading.
So you can see what is trying to be loaded
*/
app.use(function(req, res, next) {
console.log('received from', req.get('X-Forwarded-For'), ':', req.method, req.originalUrl);
next();
});
/*
Setup express Static to server those files
*/
app.use('/extension/', express.static(__dirname + '/build/'));

This will raise an express static server on port 8050, and then prepare to host the contents of build on the route extension.

So this will give us a URL of http://127.0.0.1/extension/ and if you remember in Part 3, we wrote about the structure of a URL of a Hosted test/Live extension being https://ClientID.ext-twitch.tv/ClientID/md5/yourHTML instantly our Development Environment is closer to the Production Environment.

To further this, I like to put my views into different folders. So the viewer will be in panel or video and if I offer both I’ll have both. The Config will be in config or something random for extra security on private Extensions. And Mobile in mobile if I need to serve different JS to the user.

Which then makes it even easier or a developer to remember to use relative links to their CSS/JS from the HTML, since my views are in sub folders, and the whole Dev Server is serving from a sub folder.

But what about the rest of the file? That is a basic Folder watcher, using Chokiar, that will watch for any change in the develop/dev directly and then run script.sh

This script will

  • dump the current contents of build,
  • copy the folder structure
  • copy over any “common” assets in the assets folder (background images/icons for example)
  • copy over each HTML file, in some cases run a minify process
  • compile each JS file and CSS file together into one file and run it thru minifies (but not mangles*)

*Twitch disallows manglification, except in some super limited cases

The script will call the NPM globally installed instances of:

  • html-minifier (not in this example but I use it on occasion)
  • uglify-es which provides uglifyjs
  • uglicfycss

I like to build different parts of my extensions into different script/css files and then use my develop/build process to combine them into one file. Here is FlightSimTrack’s current layout for example. Left being the built/compiled and right being the Development version.

FlightSimTracks structure
FlightSimTracks structure

You can see how my many JS/CSS on the right are folded down into singular files. And make it easy to include CSS Resets/grid systems into each view when loading/merging those files from a common folder, which only exists on the right/develop side.

FlightSimTrack, for example, has a few parts, such as

  • the maps,
  • the player information
  • Twitch Auth and PubSub handler

Which I’ve split into three files for ease of reading and modification, you can use one mega JS file or whatever compilation method you want, or not at all an include many script files! You just need to avoid magnification.

The only difference between my script.sh and my build.sh is build will generally HTML Minify where script doesn’t and build will compile the JS and drop and console.log commands, they don’t work on a released extension (and are disallowed by policy), so you may as well drop them from the files to keep the file size down! Great for Mobile users.

Summary

This will then give us a Development server, running on a Sub Folder, with files similar to what you would use in production. So this should be analogous to the Production result for your Extension.

Just one more thing

We forgot one thing, what about SSL? Oh that old chestnut! The final piece of the puzzle!

There are two easy ways to provide SSL Termination, both have their nuances but I prefer the second.

Method 1

NGROK, is a Free (or paid for product), that will create a temporary public URL to a running service on your machine.

So in this example you’d just do ./ngrok http 8050 and then the UI will display a URL to copy/paste into the Twitch Developer Console for your “Testing Base URI” just remember to add /extension/ to the end, since that is the mount point for your build. And now you have SSL Termination!

The Dev Console configured with a NGROK URL
The Dev Console configured with a NGROK URL

NGROK may have some other funnies such as rate limits, but for current limits please refer to their website and pricing structures.

Method 2

This is my preferred method, instead of using NGROK (or paying for a constant URL with NGROK).

I use a reverse SSH Tunnel, and get NGINX on a server to handle SSL Termination with a “real” free from LetsEncrypt Certificate.

Setup is the same on the user side, instead of running ngrok I ssh -R 8050:127.0.0.1:8050 username@example.com

This means I never have to update the Developer Console with a new URL, and for testing purposes all my Extensions use the Same URL. I just change the server running at the end of the tunnel. And if I start work on a new extension, I can use the exact same hosting settings.

NGINX is configured to do the normal SSL Termination stuff, then I just proxypass. Here is a config example from my live server that handles my Extension hosting.

server {
    listen someip:443;
    listen [::]:443;

    server_name example.com;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    include /etc/letsencrypt/options-ssl-nginx.conf;

    resolver_timeout 10s;

    ssl_dhparam /etc/nginx/dhparam.pem;

    location / {
        proxy_pass http://overssh8050;

           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection 'upgrade';
           proxy_set_header Host $host;
           proxy_cache_bypass $http_upgrade;
           proxy_redirect off;
           proxy_http_version 1.1;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
    }
}

upstream overssh8050 {
    server 127.0.0.1:8050;
    keepalive 8;
}

I’ll usually use a second port/SSH tunnel/SSL’ed domain to talk to my EBS running locally. And my script.sh/build.sh can be configured to use different EBS URL’s in the fetch commands you may do. One less thing to forget to swap when building for release/review queuing.

Summary, for real

That is it for this weeks post, you can have a poke about in the GitHub Repository at BarryCarlyon/twitch_extension_blog_series both for the Server.js and script files and the folder structure.

Now you should be able to setup a local test server, that is similar in URL structure to a released Twitch Extension, and provide SSL to that test server, so you can test the Extension on Twitch, OR in the Rig, two of the most common pitfalls Developers face when starting to build extensions.

BUT MOTHER I CRAVE VIOLENCE

Well, until I write the next part if you want to read more about the Developer Side of Extensions, you can pop a visit over the to the Documentation or take a look at Twitch’s Introductory Page and you can always join us on the “TwitchDev Discord Server”, visit the Developer Support Page for the current invite link!

Why you think you are good enough to even write blog posts on Extensions? I made a one or two of them Extensions of various types.

Twitch Extensions: Part 1 – An Introduction

This is the first part of a I don’t know how many parts series on Twitch Extensions. We’ll cover how to develop them, how to EBS them, some notes on Designing and what to avoid, some practicies for after release care, and touch on some marketing ideas/things for your Extension Website.

So since this is the first part, we’ll cover the basics first. What, Where, Who, How, and Why of an Extension

[caption id="attachment_51846" align="aligncenter" width="740"]When is Gamora Why is Gamora[/caption]

This is the first part of a I don’t know how many parts series on Twitch Extensions. We’ll cover how to develop them, how to EBS them, some notes on Designing and what to avoid, some practices for after release care, and touch on some marketing ideas/things for your Extension Website.

So since this is the first part, we’ll cover the basics first. What, Where, Who, How, and Why of an Extension

When is Gamora
Why is Gamora

What is a Twitch Extension

A Twitch Extension is basically an iFrame that allows a developer to create anything they want, as long as it fits within the Guidelines set out by Twitch, and of course it’s Terms of Service and Developer Agreements

This can range from MiniGames, to QnA/Polling extensions, to Community information extensions, to game information extensions, or play with the Streamer extensions

Some examples include

  • Sound Alerts – Lets users pick a sound to play on Stream, can be free or utilize bits/channel points
  • Crowd Control – Provides plugins for a variety of games, to allow people to interact with the game, make it easier/harder for the Broadcaster
  • Cardboard.live – Lets viewers see what cards are in your current deck, and check the state of the game board, without having to spam/ask in chat
  • Borderlands 3 ECHOcast – Lets viewers check out your Borderlands 3 character, and let viewers win extra loot for their own character when the Broadcaster opens Red Chests in game
  • Detroit: Community Play – Ask the community to pick/vote on the option when a multiple choice question appears during the game play of Detroit: Become Human.
  • The Cohhilition – A community interaction extension, that provides access to various Community things (in this case for a single channel), without having the viewer leave the comfort of the Twitch page

Some of these descriptions are super simple to cover the salient points and many will do more things than my summary covers

Check out more extensions at Twitch’s own Extension Discovery

Where is a Twitch Extension

Twitch being a live streaming site, provides to the Broadcaster, a page, and that page will consist of a number of elements.

Depending on if the streamer is live or not the elements on the page will vary slightly. If the Broadcaster is live, you’ll land on the video/chat page, if the Broadcaster is not live, you’ll land on a “Home”/index style page

Now the part that we care about is the “Chat”/live view page, and on that page you will find a number of sections

An example of a Twitch Channel Page
An example of a Twitch Channel Page

  • The Video Player
  • The live Chat
  • The Stream information section – The Title and Category
  • A small about the Broadcaster section
  • The Panels section

A Twitch Extension can be added to a couple of these sections, and has 3 main (and two auxiliary) integration points.

The Main Integration points

  • Video Overlay – The Extension can cover/utilize the whole of the video player
  • Panel – The Extension appears in the panels section below the stream, and has width of 318px and a maximum height of 500px
  • Video Component – Basically a panel that appears over the video player but is locked to the right hand side of the player, it can utilize a varied amount of the player space

The “main” integration points are mutually exclusive, an Extension can only occupy any one of those slots at once

The Auxiliary Integration points

An example of a mobile extension on iOS
An example of a mobile extension on iOS

  • Mobile Panel – The Extension is available on mobile for mobile users to interact, it will replace the chat, and dimensions wise basically similar to a panel on PC (in terms of ratio), but you would have to consider landscape views on tablets as well.
  • Panel popout – Panel Extensions can be opened in a new window and can be resized by the user at will

An extension can be in one of the “main” integration points, and the mobile point.

Who is a Twitch Extension

As part of being on Twitch, extensions are able to use a number of Extension Features, as well as doing more “regular” Twitch stuff. You could run a regular chat bot that runs with your extension, the Twitch Extension Timeout with bits does this in order to run the actual timeout commands on users

So aside from the “regular” stuff like chat bots, Twitch Extensions have access to some additional features

  • Bits Support – Allow Viewers, to exchange bits (a digital good) for various “digital goods” inside Extensions, this could range from an extra vote in polling extensions, or picking a victim in “Timeout With Bits”, or a cool cloak for your character in a game. Revenue generate here is split 80/20 between the Broadcaster/Developer.
  • Subscription Support – Allows the Extension to check the subscription status of a viewer on the channel the extension is installed to, avoiding the need for the Extension Developer to get and maintain oAuth access tokens from the Broadcaster “separately” to the install process of the Extension
  • Identitiy Link – Allows viewers to “login’ to your extension, we’ll cover this more in a later post in the series
  • Chat Capabilities – Allow the Extension to send chat messages (via a HTTP POST request), usually used as a notification system to prompt viewers to perform an action in the extension, like a new poll has started, go vote, for example
  • The Configuration Service – We’ll cover this in a later post as well, but it’s a way to store data on Twitch’s server that you can use in the extension, this might be something like, the name of the Broadcasters Character in a game that you would use in an API request to get information about the character
  • Streamer Allowlist – allows the Extension Developer to restrict whom can install the Extension to their channel

We’ll cover each Capability/feature in future blog posts in the series

How is a Twitch Extension

We’ll cover this more, in depth in later posts, a Twitch Extension is a bundle of files uploaded to the Extension CDN (Content Delivery Network). This needs to include your HTML, JS, CSS, and any static images you want to store on the CDN (pretty handy for background images for panel extensions). Twitch has some restrictions on what an extension can load from external sources, but essentially images are fine, CSS/JS is not, CSS/JS must be local/included.

Those files are uploaded to a sub domain of Twitch, into a particular sub folder tree on that sub domain, which we will cover more in depth in a later post, when we talk about building extensions and a suitable way to test them and some related gotchas.

All Twitch Extensions have their bundles uploaded to the Twitch and before they are released (or updated) to the masses, the Twitch Extension Review team will review the Extension, to ensure it works as intended, there is no major bugs effecting activation, the Extension compiles with the Guidelines and Terms of Services, and most importantly contains nothing malicious to interfere with the Twitch website or the viewer using the extension Computer/device

A Twitch Extension is allowed to communicate offsite, the resource just has to be secured over SSL, this is commonly referred to as an EBS or Extension Backend Service, we’ll cover this more later as well!

Why is a Twitch Extensions?

But Why is a Twitch Extension (any use) I hear you cry?

A Twitch Extension provides ways for the Streamers Community to perform rich interactions, without leaving the Twitch Broadcasters page, which means you keep the Viewer watching the Stream or interacting with Chat, with relatively easy access to Twitch API’s, without long additional steps for Viewer Authentication

Summary

That is it for Part 1 in this series on Twitch Extensions, I’ve cover the basic What, Where, Who, How of Twitch Extensions

Parts will either be weekly or bi-weekly, we will see how we go!

BUT MOTHER I CRAVE VIOLENCE

Well, until I write the next part if you want to read more about the Developer Side of Extensions, you can pop a visit over the to the Documentation or take a look at Twitch’s Introductory Page and you can always join us on the “TwitchDev Discord Server”, visit the Developer Support Page for the current invite link!

Were the sub headings supposed to make sense? No not really.

Why you think you are good enough to even write blog posts on Extensions? I made a one or two of them Extensions of various types.

How does Twitch’s new EventSub work?

Preamble

In the beginning, Twitch created Webhooks. And the world rejoiced, finally we no longer have to long poll for things such as followers! Hurrah! But it came with a gotcha, since Twitch Webhooks is based on the Websub specification, each and every Webhook you wanted to listen to had to be renewed, now this is limited to 10 days (max) or the time remainging on the Token you authenticate with (when asking to listen to priviledge topics such as subscribers), which generally meant you had to remake certain subscriptions every 4 hours, after of course renewing the token with the refresh token. Since the maximum length of a Twitch user token is 4 hours. (Implicit tokens use 60 days, but you can’t refresh those).

This wasn’t a good solution for developers, since it creates siginificant load, and “wasting” of your rate limits with Helix. Even more so when working with multiiple streamers.

Twitch has acknowledged this as a problem, which leads us to the new product of EventSub.

Aside from ignoring the need to remake your subscriptions periodically, EventSub also doesn’t need a correspoding Helix endpoint to exist for the topic you wish to listen on (this also means a different format for the data payloads), and means EventSub can make new topic types without waiting for Helix to create the endpoint first.

So, now onwards to the meat of this post!

EventSub, hows does it work?

The long and short of it, is that everything is sent via HTTP Post requests, to your SSL protected endpoint.

Unlike Webhooks, Eventsub only authenticates using App Access Tokens (for server to server requests), but how does this work when you are attempting subscribe to a channels new Subscribers topic, or the Ban/timeout events topics?

Authentication

Well, when you make a subscription request to EventSub, Twitch looks at your App Access token, then checks in the background if the requested broadcaster has connected to your Application at any point, with the relevant scopes, and not revoked that connection.

So, it’s a “two legged” approach to authentication. But means that you, the developer, don’t need to store the broadcasters access or refresh tokens, after they have authorised. though you generally would in order to perform “catch up” if your application goes offline/restarts for any reason.

Infrastructure

So to utilise EventSub, you need to create a portal that allows broadcasters to grant access to your Application to their account with the relevant scopes, using “regular” User Authentication.

Then on authentication you check/create eventsub subscriptions as needed, using your App Access Token. You should also store and use the returned User Acces token for use on catchups

And you need a server that can recieve HTTP Posts from Twitch for verification of the connection and accepting data payloads. (Generlly here you’ll HTTP 2xx OK as quickly as possible and send the data into background processes.

TLDR: for example if you can call Get Broadcaster Subscriptions (at the moment you obtained the Access Token) then your ClientID can subscribe to the channel.subscribe EventSub.

Summary

Hopefully this should help out anyone that is new to EventSub or getting involved with the Twitch Channel Points Hackathon.

Since many people may look at EventSub to recieve new Channel point redemptions but might trip up over the initial authentication setup.

There are a number of examples on my GitHub that might help, but if you need any further help, please join us on the TwitchDev Discord Server!

Just thought I would write up this post with a ramble of notes on how EventSub Authentication works, since theres gonna be a spike in people asking and probably getting stuck if they are new to working with Twitch.

Twitch API Examples

I spend a lot of time on the Twitch Developer forums and Discord helping out other third party developers. That among other things led to me being asked to become a Twitch Ambassador, which is probably a story for another post.

As part of spending a lot of time helping of Forums/Discord, it become useful to write up some examples in various languages for people to refer to, since some people prefer code examples over documentation, and it’s easier to demonstrate how to tie multiple calls/endpoints together for the desired result.

To that end my GitHub Repo at barrycarlyon/twitch_misc now exists and holds examples from Authentication flows (from Implicit to server access and regular user in-between), extension config/pubsub, and examples for Webhooks and the new Eventsub (which is worth a look!). So if you are looking for some examples do checkout the Repository. Some of the examples can even be tested on GitHub itself via GitHub pages, the examples available are listed in the readme and at the Github Pages site.

Twitch also recently made the requirement that all calls to helix (aka the New API) need to be Authenticated using a Bearer, which made it difficult for Extensions to get the viewers details. So to that end I created a basic example of how to do that in an Extension with a “User Profile Extension” example. Which is at BarryCarlyon/twitch_profile_extension. So this covers a good way to handle that flow.

Right now most of the examples are nodeJS, or PHP, but there are some in Python kicking about!

I’ll be looking at adding more examples and other examples in other languages as we go!

I’m usually really bad at commenting my code as I prefer reading the code, but I made a conscious effort to add useful code comments on these repos!

The Future and all things change?!

Long time no write.

Hopefully gonna start writing again, (not the first time I’ve said that, so we’ll see how we go!)

Anywho, I do currently find myself unemployed after quitting my previous Job on Monday, nothing on my part but the actions of others leading to the actions of others forcing my hand.

I am currently deciding what to do next!

If you know whom I used to work for you can go look it up yourself, I don’t feel the need to talk about it myself.

I am continuing to do various bits and pieces for the Streamers I work with on Twitch, so nothing has changed there, I’m just out of full time work!

Watch this space is the TLDR!

In the mean time, go checkout my Twitch Extensions