The End of Twitch PubSub

Finally Twitch EventSub is at feature parity with third party supported/documented PubSub, that the end of PubSub has been announced.

In a forum post over on Twitch Developers it has been announced the schedule for the shutdown of Twitch PubSub for access by third party developers.

The final death date is April 14th 2025.

Which I guess means it’s time for me to complete my own migration from PubSub to EventSub. Which at least is JUST ABOUT planned….

At least for 90% of what I need it’s ALL USERNOTICE as an EventSub topic, which is this case is channel.chat.notification and then whatever I choose to go with for consuming cheering.

So then for anything else I’ll need that doesn’t exist, either from it not existing in the first place OR a undocumented topic I might be using, off to uservoice I’ll go!

Off hand I think there is only one thing but we’ll see when I work on and complete my migration! The only other barrier is gonna be authentication but I have authentication or can get it for everything I need.

It is worth noting that Extension PubSub is unaffected, as that for the most part is a completely different system that just shares the same name.

Topic Migration notes

Edit: I guess I should add some notes about migration

PubSubEventSub
channel-bits-events-v1.<channel_id>channel.cheer or channel.chat.message
channel-bits-events-v2.<channel_id>channel.cheer or channel.chat.message
channel-bits-badge-unlocks.<channel_id>No Route, or inferred from channel.cheer
channel-points-channel-v1.<channel_id>channel.channel_points_custom_reward.add
channel-subscribe-events-v1.<channel_id>channel.chat.notification is what I will use for my use case of overlay/on stream alerts. Other type segregated topics exist
automod-queue.<moderator_id>.<channel_id>automod.message.hold
chat_moderator_actions.<user_id>.<channel_id>channel.moderate
low-trust-users.<channel_id>.<suspicious_user_id>automod.message.hold and automod.message.update
user-moderation-notifications.<current_user_id>.<channel_id>channel.chat.user_message_hold
whispers.<user_id>user.whisper.message

Twitch Extension Developer Tools

Around the time that Kraken was dying, I was doing something (I forget what, probably bits products), and found that the Twitch Developer Rig was totally hosed, since it was calling Kraken and hadn’t been patched. (It since has been but still being a little weird with some stuff with some of my Extensions, but I digress).

So I was like “screw it I’ll build my own tool with Blackjack and Hookers”.

So I went and built my own tool, it’s part “API testing tool” part “useful things for managing Extensions”. And released Open Source over on GitHub under the WTFPL aka – “Do What the F*ck you Want to Public License”.

Main reason for making a tool rather than a bunch of test scripts, was I wanted to easily store and recall my Extension Client ID’s Extension secrets, and easily generate App Access Tokens for the relevant API’s that needed.

So without further ado, you can find, download, use and abuse, the tool over on Github, or even PR a feature in https://github.com/barrycarlyon/twitch_extension_tools current Version is v1.1.3!

It should offer just about every “useful feature” from the Developer Rig, that there is a public API for, except for “Views” and “Start my servers for me”. For a easy summary of the Features Available, checkout the Readme on the GitHub!

DEATH TO KRAKEN

Todays the day!

Twitch’s Kraken API gets turned off in checks watch just over three hours!

I know I’m just about ready, but if you are not, and you already missed the shutdown test windows, heres some notes!

There is a v5 Migration guide – https://dev.twitch.tv/docs/api/migration

DO NOT FORGET this also effects the Extension API endpoints (such as Send PubSub message and Send Extension Chat message), this doesn’t affect the Extension JS Helper!

For most server to server operations you can use a Client Credentials Access Token

Most people will likely be using Ajax calls in website front ends and are now confused how to migrate, there are two routes for this problem. You either ask the user to login with Twitch to provide a token to use (this is how my GitHub examples work) but not very user friendly, you don’t want to prompt the user to login to your website JUST to show them if your stream is live or not. The second route is your frontend needs to call your server and the server uses a Client Credentials token to call the API with.

Alternatively if you are doing something “simple” such as “check my stream is live” then you should consider moving from API calls to EventSub instead. Then when a channel changes Title or stream status, Twitch will tell you in (near) real time. Collect and store that data in a database, and now your website doesn’t need to call the Twitch API each time someone loads your website.

For further help migrating or to join us for the DEATH TO KRAKEN party (bring your own confetti) join us on the Twitch Dev Third Party Developers Discord you can find a link on the Twitch Dev Support Page

For reference here is the Dev Forums link with the shutdown window timings: https://discuss.dev.twitch.tv/t/legacy-twitch-api-v5-i-e-kraken-shutdown-reminder-february-28-2022/36589 the shutdown originally announced back in July.

If a feature is missing or theres something you need thats not in Helix, don’t forget to file a UserVoice. I know a lot of people need/want Clip Offsets so make your voice heard with UserVoices and votes on existing issues. See also my mega UserVoice sheet. And checkout the GitHub for known issues.

Twitch Extensions Part 6 – Dev Environment Updates – Content Security Policy!

In part 5 we wrote about a suitable testing platform for building your extensions on, essentially we create a static content server, that mimics the Twitch CDN for testing with.

Twitch Announced on the Forums that they are revising the CSP (Content Security Policy) that extensions use to protect and control what can be loaded. I wrote about this in the previous blog post.

I’m currently waiting on a response from Twitch (via the forums) about any other changes to the CSP, but for now, you can test the changes today!

What Even is CSP

First lets do a quick explanation of what CSP, CSP is Content Security Policy, a browser technology to help control what a given Website can load and what browser functions are allowed.

The HTTP Content-Security-Policy response header allows web site administrators to control resources the user agent is allowed to load for a given page. With a few exceptions, policies mostly involve specifying server origins and script endpoints. This helps guard against cross-site scripting attacks (Cross-site_scripting).

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy

You can read more about CSP and the various things it can do over on the MDN Web Docs. There is a lot more that can and can’t be done with CSP more than just controlling what content can be loaded from where, but for Twitch Extensions we only need to consider the parts of the Policy that affect Twitch Extensions.

Twitch Extension CSP Policy

Twitch is requiring Extension developer to declare the Connect, Img, and Media domains, which in the policy are connect-src, img-src and media-src. You can declare this in the Developer Console for a version of your extension, under the Capabilities tab.

The New Extension Dashboard fields
The new Extension Dashboard fields

Now, the items you enter here only apply when you are using Hosted Test (or release), since Hosted Test will use Twitch’s CDN, and thus Twitch’s Server which can load and use the relevant fields, but in localtesting (aka not the CDN) we need to set this up ourselves.

Local Testing a CSP

If you have been following this series, then you already have a Node/Express server that will run a static output for you. We can easily add CSP headers to this server using a module called Helmet, generally speaking it’s wise to consider adding helmet (or CSP Headers in general) to any website you run to protect your users, but I digress!

So, how to set this up for Testing with.

Normally I’d say, on server start call the API to get the current extension settings from the console, however, the API at this time has not been updated to include the new fields, I raised a UserVoice requesting the new fields be added to the endpoints. And you can upvote that here.

So for now, we’ll need to populate the CSP for Helmet manually.

Configuring Helmet for CSP

The first thing I did was look at a released extension to see what the current CSP is, which I then split out into a object for configuring Helmet with. Then I looked at what the rig needs, and then looked at what you need to add to correctly simulate a CSP.

The base CSP for a Twitch Extension is, here twitch.client_id is loaded from an external config file, and represents the location that Hosted Test and Release use to host your files. Which I’ll touch on later.

/*
Current base CSP rules subject to change

See:
https://discuss.dev.twitch.tv/t/new-extensions-policy-for-content-security-policy-csp-directives-and-timeline-for-enforcement/33695/2

This example is based off a live extension
*/

let contentSecurityPolicy = {
    directives: {
        defaultSrc: [
            "'self'",
            `https://${twitch.client_id}.ext-twitch.tv`
        ],
        connectSrc: [
            "'self'",
            `https://${twitch.client_id}.ext-twitch.tv`,
            'https://extension-files.twitch.tv',
            'https://www.google-analytics.com',
            'https://stats.g.doubleclick.net'
        ],
        fontSrc:    [
            "'self'",
            `https://${twitch.client_id}.ext-twitch.tv`,
            'https://fonts.googleapis.com',
            'https://fonts.gstatic.com'
        ],
        imgSrc:     [
            "'self'",
            'data:',
            'blob:'
        ],
        mediaSrc:   [
            "'self'",
            'data:',
            'blob:'
        ],
        scriptSrc:  [
            "'self'",
            `https://${twitch.client_id}.ext-twitch.tv`,
            'https://extension-files.twitch.tv',
            'https://www.google-analytics.com',
            'https://stats.g.doubleclick.net'
        ],
        styleSrc:   [
            "'self'",
            "'unsafe-inline'",
            `https://${twitch.client_id}.ext-twitch.tv`,
            'https://fonts.googleapis.com'
        ],

        frameAncestors: [
            'https://supervisor.ext-twitch.tv',
            'https://extension-files.twitch.tv',
            'https://*.twitch.tv',
            'https://*.twitch.tech',
            'https://localhost.twitch.tv:*',
            'https://localhost.twitch.tech:*',
            'http://localhost.rig.twitch.tv:*'
        ]
    }
}

const helmet = require('helmet');
/*
You can use Security Headers to test your server, if this server is web accessible
https://securityheaders.com/
It'll test that your CSP is valid.
Best testing done with an extension, on Twitch or in the rig!
*/

console.log('Going to use the following CSP', contentSecurityPolicy);

app.use(helmet({
    contentSecurityPolicy
}));

This I add after app.listen and before anything else! It does need to go before your app.use for express.static

This will configure your test server to use the base/default CSP. And will log it out the full CSP to the console when you start the server.

The Extension Developer Rig

So the next step is how to enable your test server to work in the Twitch Extension Developer Rig. I don’t often use the rig, but it’s handy for spot testing views and mobile when I don’t have my phone handy (or the Extension has not been iOS allow listed yet!)

The Extension Rig is built in Electron, which means it will include calls to file and in testing it spot calls some other things.

For the rig I add the following rules, which I append to the default CSP using a Config Switch.

/*
should we enable the Rig?

The rig being an electron app, will call some other things
As well as having a file:// based parent
*/
if (csp_options.enable_rig) {
    let rig_sources = {
        connectSrc: [
            'wss://pubsub-edge.twitch.tv'
        ],
        frameAncestors: [
            'http://localhost:*',
            'file://*',
            'filesystem:'
        ]
    }

    // append these to the CSP
    for (let sourceType in rig_sources) {
        for (let x=0;x<rig_sources[sourceType].length;x++) {
            contentSecurityPolicy.directives[sourceType].push(rig_sources[sourceType][x]);
        }
    }
}

Nothing to silly there, but important if you are testing in the rig. Only enable this in your server when rig testing not testing on the Twitch website, as it’s overly permissive and might catch you out later.

My Sources

The final thing to do is to setup your sources, now this gets a little weird, as a valid CSP rule can omit the schema of the URL (see note).

For this example/setup we are adding the content domains to all three CSP directives. Using this example you can adjust and modify this as granularly as you want.

/*
Did we configure places that we can/may load media from
And yes we are just gonna glob them to all three groups
For example purposes
*/
csp_options.content_domains.forEach(domain => {
    contentSecurityPolicy.directives.imgSrc.push(domain);
    contentSecurityPolicy.directives.mediaSrc.push(domain);
    contentSecurityPolicy.directives.connectSrc.push(domain);
});

Note: In testing browsers will not enable/allow WSS if you declare a schema-less domain of www.example.com. So if you want WSS you need to declare it explicitly, for this I declare wss://www.example.com and https://www.example.com in the rule (not the lack of a trailing /).

I configure these schema+domains in an external configuration file for the server. Here is an example config.json:

{
    "listen": 8050,

    "csp_options": {
        "enable_rig": true,
        "report_uri": false,
        "ebs_domain": "myebs.com",
        "content_domains": [
            "https://mywebsite.com",
            "wss://mywebsite.com"
        ]
    },

    "twitch": {
        "client_id": "abcdefg"
    }
}

EBS?

If your extension utilizes an EBS you’ll need to declare that and add it to your connect-src, however if you also load images from your EBS you can skip this step.

I generally put my images and assets on a seperate server to my EBS, but for test purposes, this server example adds the EBS domain to all three declarations, for both schemas:

/*
Did we configure an EBS to call
*/
if (csp_options.ebs_domain) {
    console.log('Appending EBS Domain');
    let ebs_rules = {
        imgSrc: [
            'https://' + csp_options.ebs_domain,
            'wss://' + csp_options.ebs_domain
        ],
        mediaSrc: [
            'https://' + csp_options.ebs_domain,
            'wss://' + csp_options.ebs_domain
        ],
        connectSrc: [
            'https://' + csp_options.ebs_domain,
            'wss://' + csp_options.ebs_domain
        ]
    }

    for (let sourceType in ebs_rules) {
        for (let x=0;x<ebs_rules[sourceType].length;x++) {
            contentSecurityPolicy.directives[sourceType].push(ebs_rules[sourceType][x]);
        }
    }
}

Full Example!

I put all of this together as a full example over on my GitHub. See Part 6 of the repository. This provides a “rig” as described in Part 5 but with the additional CSP Fields included.

To set this up do as follows

  • Download the Example from GitHub
  • Copy config_sample.json to config.json
  • Populate the twitch->client_id with your Extension Client ID
  • Revise the listen port if needed
  • Configure Your CSP options as needed, add the content domains as needed. And you EBS domain as needed.
  • If you load content from your EBS domain, set the ebs_domain to false, to avoid a duplicate declaration of a domain, or do not include your ebs_domain in your content_domains

Once you have setup the server, you can the test your Rig via Security Headers which will test that your CSP is valid, however this only works if your Test Server is accessible from the internet! Which if you follow Part 5’s note will be for SSL testing purposes! And will only test that your CSP looks correct, not that it functions as intended!

Then you can move on to testing your Extension and check that your CSP works as intended, then you do not have to move to hosted test and back to test changes to your CSP!

If/when the API is updated to return the new fields, I’ll add a part 6.5 (probably) which will use the API to get the details instead. Sods law you’ll add a domain to your Test Rig/Server, and then forget to add the same domain to your Capabilities tab!

DEADLINE

Twitch will begin to Enforce the new CSP policy on January 25th.

Twitch closes for the holidays between Friday, 12/17/21 – Monday, 1/3/22. Twitch requests that Developers submit their extensions for review no later than Wednesday, 15th of December at 3PM PST.

Upcoming Winter Break

Attention, developers! Please note that the review team will be observing a winter holiday break from Friday, 12/17/21 – Monday, 1/3/22 and will not be performing Organization, Game, Chatbot Verification, or Extension reviews during this period of time. If you need a review completed prior to the holiday break, please submit your review request by no later than Wednesday, 12/15 at 3PM PST. Thank you for your understanding & happy holidays!

From the Twitch Developers Console

ONE MORE THING: Report URI

Well what about the report_uri, that you saw in the config.json example?

Well CSP provides a method to report CSP errors to a defined HTTPS POST endpoint. So whenever a CSP error occurs it can be reported to that HTTPS URL, very handy to help debug issues.

So if you configure your report_uri to be the same URL as your Extension Test rig, but with /csp/ on the end, so if your rig is at https://mytestrig.com/ then your CSP Report URI is https://mytestrig.com/csp/

You can capture and log these reports, for Express you will need to use the following code snippet, please note that a JSON payload is posted but using an alternative content-type, so you need to tell express.json to trigger on that content-type of application/csp-report

/*
This will capture any CSP Report and dump log it to console
*/
app.post('/csp/', express.json({
    type: 'application/csp-report'
}), (req,res) => {
    console.log(req.body);

    res.send('Ok');
});

The report-uri documentation over on MDN includes a PHP example, if that is your cup of tea!

An Update! Easier Development!

Added a small update to this post for easier testing with, first I took the entire CSP component and seperated it into a NPM module for easier usage and configuration.

The module, `twitchextensioncsp` can be found over on NPM and on GitHub and essentially just wraps Helmet for you and passes in the CSP Configuration with much less copy/paste between extensions if you are working on multiples.

And the Example simple server (remove the “build” system) is at Part 6.5 on the GitHub Repository

For “ease” of use heres an “quick” static Express Server implementing the module, it will do the following:

  • Create an Express Server on port 8050
  • Invoke twitchextensioncsp
  • Enable the Extension CSP to support the Twitch Extensions Rig
  • Add Img and Media and connect example domains
  • Static mount the build directory onto extension so your testing base URI is http://localhost:8050/extension/ swap as needed depending on your SSL solution
const express = require('express');
const app = express();

app.listen(8050, function () {
    console.log('booted express on 8050');
});

const twitchextensioncsp = require('twitchextensioncsp');
app.use(twitchextensioncsp({
    clientID: 'abcdefg123456'
    enableRig: true,
    imgSrc: [
        'https://images.example.com'
    ],
    mediaSrc: [
        'https://videos.example.com'
    ],
    connectSrc: [
        'https://api.example.com'
    ]
}));

app.use('/extension/', express.static(__dirname + '/build/'));

If you refer to the README for twitchextensioncsp there are a handful of quick start examples for the CSP setup. As you do need to explicitly declare the Twitch CDN and Twitch API if you wish to use those in your Extension frontend!

Twitch’s Extension CSP is changing!

Well it looks like Twitch is updating the rules around extensions! It’s a good change even if I do need do an update to pretty much EVERY single Extension I run….

You can read the details on the change over on the Twitch Developer forums but here is the long and short of it how I see it.

IP GRABBERS?

There has been a recent spate of “attacks” related to Extensions where a “compromised Extension”, allows “not the developer of the extension” to obtain the IP Address of an Extension User/Viewer. And it’s only made it onto the Twitch Reddit as it was continually abused when discovered rather than responsbily disclosed to Twitch.

Side Note: You can find out about Security and Responsible Disclosure over on https://www.twitch.tv/p/en/security/.

You find a problem?

Don't abuse it, report it responsibly and give time for Twitch to fix the problem before you go public.

The same applies when Disclosing Security issues to any site or tool, they will have a method to report security issues, and generally you give time for them to resolve before going public.

For all intents and purposes an Extension is just a Website, sat in an iFrame on a Twitch page, that could be on top of the player or below the stream.

Most Extensions usually involve calling an API or loading External images. This API and image storage service will generally be controlled and owned by the Extension Developer.

So for example, the IGDB.com Game information makes a call to my Extension Backend Service, which then proxies the request to IGDB’s API for data and checks a cache so I’m not hammering the IGDB API every second.

Side Note: In this case IGDB's API requires this route, as the API can't be called directly due to the Authentication Token requirement.

This means that a viewers IP address is provided to my server. And this is noted in my Privacy Policy for the Extension, and I’ll use this to spot and defend against bad actors or malicious users. This is essentially how all websites work. You visit a website and your IP Address is collected by that website in it’s access logs, and considered for defence if malicious traffic occurs.

The IGDB extension will show images from IGDB’s CDN for cover art and screenshots, which means the users IP address is then passed to IGDB’s CDN, for the same tracking purposes.

The IGDB extension is configured to only allow images to be displayed from IGDB’s CDN or Twitch’s CDN (when I defer to using Twitch Cover art instead).

Now a given extension could directly call an API rather than going via an EBS/Server. Which means the IP Address may get collected by the API Logging. I avoid this route as I prefer to call my server and cache data, why go and get a players Game Inventory from the Game API if I already collected it 10 seconds ago? So save some time and use a cache, making a better experience for the Viewer/User of the Extension.

Now the “Exploit”

Now the “exploit” that has been occuring is from Extensions that allowed the broadcaster to add an image to the Extension to customise or theme it to their channel.

And what developers of these Extensions did here was let the broadcaster enter ANY URL. Which meant that an attacker could put an image URL to anything.

In this case, generally the old forum signature style PHP Image scripts that would do the “Hey you from Country” where the Country is GEOIP’ed from the users IP Address.

Essentially this is just a Privacy Policy violation, and nothing to be too worried about (broadly speaking) since the ONLY INFORMATION THE ATTACKER GETS is the IP Address and basic browser information (ie, you have chrome or edge etc)

Personally if I was allowing broadcaster customisation using images, I’d have broadcasters uploading images to my server and my server then handles hosting and distribution. Which means I can then optimise that distribution of content (why send a huge image to mobile when I can auto resize and send a mobile optimised image) and plug this obvious “data leak”. And means my Privacy Policies are easier to write for GDPR compliance/etc, when I don’t have to say “data goes anywhere as I allow any hosting service”.

But Developers may avoid this as it means that their free to run extension now costs money to run. Using the config service to store config, which is configured from the config view all internal to Twitch using the Extension JS helper, for example. Then allow an external image from Imgur or any other Hosting provider. (This being the loophole/exploit)

But What can an attacker do?

Not a lot. Generally just scaremongering.

Barry Carlyon, yes quote me on this!

Sure, an IP Address can be used to obtain an approximate location of the person using that IP Address, but it’s often wrong (quite often my IP resolves to North Wales, and I’m sure as hell not in North Wales, which is funny when I get Spotify ads in Welsh).

And sure an attacker might throw out a DDoS Attack, but an ISP will mitigate that, and potentially take additional actions against the Attacker.

And in all cases, all they have is an IP Address, they don’t know whose IP Address they collected.

What is Twitch Doing?

2.12 You must provide all URLs that are fetched by the Extension front end on each version submission, this includes but is not limited to images, video, audio, and fetch/XHR requests.

Extensions Guidelines and Policies

Twitch is updating the Extensions Guidelines and Policies with Policy 2.12, included in full above.

Extension Developers are required to provide a list of domains that the extension will call, both for “What API domains are you calling” and “what image asset domains are you calling”.

It is an expansion on the CSP that was already in place, that prevents offsite JS and inline JS being executed.

So Twitch will now also provide CSP Directives for img-src, media-src and connect-src.

You can read about Content Security Policy over on MDN Web Docs. But heres the three items in a quick summary, please make sure to visit the MDN documentation for further reading!

connect-src

Connect Source controls the URL’s that script interfaces can call. So this would be things that you would JavaScript fetch against or similar. Basically the offsite API that you would call.

media-src and img-src

This will control and limit the domains and URL’s that <img and <media or JavaScript new Audio can load content from, for a brief list of examples.

In addition to this make sure to check out the Restrictions on content section of the Extensions Documentation for further reading on other limitations and restrictions!

Developers will be able to provide this domains and URLs, under the “Capabilities” tab in the Extensions Dashboard for a Version of an Extension. Heres a screenshot of the new fields, the placeholder test shows full domains including schema, but schema list domains are accepted. (Refer to the MDN Web Docs)

The three new fields at the bottom of Capabilities allow Developers to define the CSP for their Extension.
The three new fields at the bottom of Capabilities allow Developers to define the CSP for their Extension.

More on Content Security Policy

See MDN’s introductory article on Content Security Policy.

In Conclusion

It’s a good change, and will help improve security for Streamers, Developers and Viewers alike.

I will be updating my blog series with a new Entry when I’ve updated my local test rigs to help simulate and test this new requirement! It should be a relatively straight forward thing to implement as I’ve been working with CSP’s on Websites I’ve been building to improve security and prevent CSRF attacks and the like.

January 25th 2022

Developers have until January 25th to get their updates in as that is when the CSP revisions go into place!