As Report URI has continued to grow, we're constantly hearing about new things that our users want. Alongside maintaining the site, fixing bugs and constantly scaling, we're always listening to feedback to guide us on feature development too. Based on what we've heard, we got to work on the next set of features and we're now ready to share them with you!
The CSP Wizard
As a CSP reporting service, our goal was simple, to allow people to collect reports! The problem we quickly found though was that it was quite difficult for website operators to get to a point where they had a CSP they could deploy and actually, they needed help creating the CSP first, before they could send reports. This was what drove the creation of the CSP Wizard back in 2018.
Since its inception, the CSP Wizard remains one of the most used features outside of our core service offering of gathering reports and has helped countless websites quickly and easily get most of the groundwork done for creating a viable CSP. As use of the Wizard has grown, and as it is used by larger and more complex organisations, a common request has been for more flexibility in how the Wizard can be used, which is our first new feature!
You've always been able to have multiple CSPs in the 'My Policies' section of your account, but you could only ever work on one of them at a time in the Wizard. Our idea was that you would do the bulk of the work for a policy on a particular site and once ready, you could finalise the policy for deployment and move onto the next site. What we've found over time is that organisations that operate a large number of sites, or a few sites that are very complex, may sometimes want to switch and change between the policy they're currently working on. Our Teams feature does allow this, because each individual team can work on their own CSP for their own site, but the team could still only work on a single policy at once. No more!
When switching between the policy that is currently being used with the Wizard, no progress will be lost in the Wizard. Previously, when switching between the policy that is active with the Wizard, our view was that work on this policy was 'complete' and the Wizard items would be cleared, leaving you with your final CSP. It turns out this wasn't the case and required a little bit of re-architecting on our side for users or teams to now be able to dynamically switch between policies being used in the Wizard at any time you like!
The CSP Builder
This tool has been present on Report URI since day one and allows you to create a Content Security Policy using a GUI-based tool. It's been really popular since launch in 2015 and has received various updates () since. The most commonly requested feature for the CSP Builder was a tighter integration between the builder and the policies users have in the My Policies section of their account, which we're now happy to announce!
The CSPs in the My Policies section of your account can now be manually edited in the CSP Builder with a single click of the new 'Edit' button.
Once opened in the CSP Builder you have all of the powerful editing features of the CSP Builder available and can make any manual changes that you might require, or even build an entire policy from scratch!
What's even better is that you can build most of a policy using the CSP Wizard and then just tweak it with those final changes using the CSP Builder if you need to. This will make the process of building a CSP even easier and you can now go from a blank policy, all the way to a fully functioning CSP in our UI without the need for any other tools or processes!
Domain Generation Algorithms
Often used by malware and bad actors, a DGA does exactly what it sounds like; it's an algorithm used to generate domains! You can read more on the DGA Wikipedia page, but this paragraph gives a great overview:
The technique was popularized by the family of worms Conficker.a and .b which, at first generated 250 domain names per day. Starting with Conficker.C, the malware would generate 50,000 domain names every day of which it would attempt to contact 500, giving an infected machine a 1% possibility of being updated every day if the malware controllers registered only one domain per day. To prevent infected computers from updating their malware, law enforcement would have needed to pre-register 50,000 new domain names every day. From the point of view of botnet owner, they only have to register one or a few domains out of the several domains that each bot would query every day.
The DGA follows a particular pattern when it comes to the format of the domain it generates, but various families of malware and other similar threats can create a wide variety of formats, making it difficult to identify them.
I first started digging into DGAs when looking at our own internal data feeds and I started to notice frequent and odd domains. One thing we track as a service is the very first time we've seen an asset blocked from a particular domain. Whilst we do process around 700,000,000+ CSP reports per day, I've talked about our use of probabilistic data structures like Bloom Filters and how we can do this kind of analysis with relative ease. If I were to give you a sample of the domains that caught my eye, you could probably guess why they did.
script-src-elem) or blocked outbound communications (
connect-src), and look for anomalies. We've built the chain from a set of ~7,000,000 known good domains and are now using it to identify DGAs, with the small selection above being some that we've identified in production data.
I plan to write a technical blog post on how we've built and used a Markov Chain to implement this new feature, but for now I want to announce that it's in open beta for all users, paid and free. At some point in the future this, and other Threat Intelligence based features, will likely be restricted to paid-only plans, but for now, whilst we continue to improve and work on them, they're available for everyone to use.
To use it, we currently have a hidden query string parameter on the Reports page. It's been there for a while as I've been testing it with various enterprise customers and we're now ready to share it with everyone. Once we're happy that we've made enough progress it will be added to the UI, but for now, simply add the
dga=on query string parameter either on its own, or along with other query filters, to turn on the DGA filter.
Indicator Of Compromise
As a service processing all of your CSP reports, we're in a great position to be able to look out for IOCs in your reports. We already have powerful features like Script Watch, which audits your JS dependencies on an ongoing basis, and Data Watch, which audits the locations your site is sending data to on an ongoing basis, but we wanted to take that a little further. Right now these features will only tell you about new scripts or new locations you're sending data to, which is great because you can immediately become aware of them and check they're expected, but we wanted to start going further.
The difficulty came in taking the step between saying "hey, you have a new JS dependency" and saying "hey, you have a new JS dependency and it's super bad!!!". To begin this process, we've now started ingesting various sources of Threat Intelligence data that we can use to enrich our analysis of inbound reports. Just like the DGA analysis above, this feature has been present for a short while now and I've been experimenting with it alongside some of our enterprise customers. It's present on the CSP Reports page and activated in a similar fashion to the DGA feature above, with an undocumented query string parameter.
You should of course always hope that any query with the
ioc=on parameter returns no reports, but if it does, it could be something worth looking at. As we continue to improve this capability, the IOC and DGA markers will become visible in other features like Script Watch and Data Watch, along with configurable alerts that can be sent immediately upon us detecting either one of these issues.
The IOC and DGA features above are just the start of what we're planning and we have a lot more coming. We're currently trialling other sources of threat intelligence data to enrich our own analysis and once we've made a little more progress, they'll be announced in open beta too. In addition we're also looking to include WHOIS data to look at the age of domain registrations because brand new domains might require more immediate attention and identifying associated domains to see if we can link a domain to other IOC data. This is just the start of some really big ideas we have and I'm excited to continue developing our capabilities in this space!