Back in 2021, Google launched, alongside other organisations, a new security baseline for products known as the Minimum Viable Secure Product. Now, 2 years later, they've released an update to that standard.

This a cross-post of my article on the Probely blog, you can read the original there.

Minimum Viable Secure Product

You can read the original announcement from Google if you like, but we'll be focusing a lot more on the update released a couple of days ago. The MVSP site is also a great place to get a lot more detail on the project and track future changes or updates.

In terms of what the MVSP project is trying to achieve, I think this snippet from the site gives a really good idea of exactly what it's about:

Minimum Viable Secure Product (MVSP) is a list of essential application security controls that should be implemented in enterprise-ready products and services. The controls are designed to be simple to implement and provide a good foundation for building secure and resilient systems and services. MVSP is based on the experience of contributors in enterprise application security and has been built with contributions from a range of companies.

I've said myself many times in the past that sometimes, we need to focus on getting the basics right before we get carried away on more complex or elaborate risk reduction, and MVSP aligns extremely well with that approach. You should read through all of the requirements outlined on the site, of course, but I'm going to pick a few that are near and dear to my heart to focus on!

§1.1 External Vulnerability Reports

As I sit and write this blog post, I'm currently going through two responsible disclosure processes where I'm desperately trying to get in touch with the organisations in question. I've tried email to customer services, I've raised a support ticket, I've reach out to people listed as employees on Linked In and finally, I have to resort to public calls for help:

Hey @capellisport, we've made several attempts to contact you via various channels!

Does anyone have a security contact or some way of reaching an appropriate person? Please reach out, my DMs are open.

— Scott Helme (@Scott_Helme) November 21, 2023

This is downright ridiculous and it should not be this hard to get in touch with an organisation to let them know that they have a serious security issue!! That's precisely what the Security.txt file is for. You can read the full details in that blog post, but the TLDR; It's a simple text file you host with details on how people can contact you to report security vulnerabilities / responsible disclosure.

You can see mine here:
https://scotthelme.co.uk/.well-known/security.txt
https://report-uri.com/.well-known/security.txt
https://securityheaders.com/.well-known/security.txt

§1.4 External Testing

No matter how good your own security processes are, you always need another set of eyes to spot the issues you've missed. As a penetration tester for many years myself, I understand the value of such services form both sides of the conversation, as the 'hacker' conducting the test, and as the company on the receiving end. My own company, Report URI, has just had its annual penetration test completed by an external company and, as always, the full report is published for anyone to see!

I don't think you can just have an annual penetration test and then brush your hands together and say 'all good', though. Penetration tests won't find all issues, and, if you're only having a test once per year, an issue could easily sit around for 6+ months until it's discovered. Not good... That's why it's also a great to idea to use a DAST solution like Probely that can scan your application for vulnerabilities on a far more regular basis.

Just like a penetration test, Probely (or any other tool), won't find all issues, but they can find issues much sooner and much cheaper than a penetration test. There's no point in spending thousands of dollars for a penetration tester to find and report issues that you could have found for hundreds of dollars instead. Not only that, but you can find them sooner, and we've all seen the graph on the cost of fixings bugs, right?

Remember that all security vulnerabilities like this are basically just bugs that need fixing! The sooner you find them, the cheaper they are to fix and the less risk you were exposed to.

§2.3 Security Headers

Yeah! Who doesn't love some Security Headers?! Not only do they recommend using Security Headers (the actual HTTP response headers), but they also recommend using Security Headers (our website!) to scan and assess your headers!

You can see the recommendation and link to us right here, and I'm super grateful for our free service to be mentioned like this. Head over to our site and you can perform a free scan that takes 2-3 seconds right now!

§3.3 Vulnerability prevention

After working in the Cyber Security world for so long, one thing that I realised was there are always solutions to any problem you may have, and often, they're quite easy. The problem that I've always come across is that people simply didn't know about the solution, and it's one of the things I focus on in both training courses I deliver. You can see full details on my Training page, but here's the summary of the two training courses I deliver:

Hack Yourself First: In collaboration with Troy Hunt, I deliver his awesome, 2-day workshop where we learn how to hack in to our demo application and then how to defend against those attacks.

Practical TLS and PKI: In collaboration with Ivan Ristic, I deliver his incredible, 2-day training where we deploy and fully configure TLS and PKI in real-world environments. This training course also ties in superbly well with MVSP §2.2 HTTPS-only and §2.8 Encryption!

Do you meet the MVSP requirements?

I'm sure that many of you reading this blog post will be able to quickly flick through the list of requirements and check them off, but can you check off all of them?

It's really interesting to see the direction that MVSP is going in and I wholeheartedly agree with everything that's in there. As the name would imply, this is the Minimum Viable Secure Product and not the Maximum Viable Secure Product, so you could and should be exceeding many of these requirements, especially if you're a security focused company like we are! I'd like to leave you with this quote from the MVSP docs, highlight my own:

Minimum Viable Secure Product (MVSP) is a list of essential application security controls that should be implemented in enterprise-ready products and services.

It's that time of year again at Report URI, right before we start getting festive, that we have our annual penetration test and 2023 is going to be our fourth test that we publish in full.

Penetration Tests

Our previous penetration tests, which were also published publicly, were back in 2020, 2021, 2022, and now it's time for the 2023 edition. Just as it was before, there were no artificial limits placed on the scope of the penetration test and it was a 5-day test. Again, we provided our source code as part of our test because it can help the tester move much more quickly and confirm or even discover issues by looking over the code. I really stressed the point that we wanted to be absolutely confident that we didn't have any issues lurking after another year of introducing new code and bug fixes, so I requested the best tester they had on staff. I also provided as much information as I could, full featured accounts, test data, a demo of the application and access to our source code.

The Results

Maintaining the same number of findings as last year, but swapping around the ratings a little, here is the summary for our 2023 findings.

This is an outcome to be really pleased with, given that we were trying our best to uncover any issues that we may have and leaves me feeling really confident that our existing processes continue to serve us well. Let's take a look through the report in more detail and see exactly what was found.

5.1 Vulnerabilities in Outdated Dependencies Detected

Again?! This one did come as a little bit of a surprise to me and you may recognise this from the 2022 report. This finding is in a different dependency that has since had an issue identified, but I was surprised that we were able to use a dependency with a known vulnerability.

We have a GitHub action that checks all of our JS dependencies for known issues but it seems it was having trouble with this one which I'm investigating further. This has now been resolved and the dependency updated!

It was also noted that our version of Bootstrap was EOL but there are currently no known issues and we have our CSP as an additional control too. This is something we're aware of and it will be addressed in due course.

5.2 Insecure TLS/SSL Configuration

This was only raised as an info item on the report and is also something we're aware of. Given the wide variety of clients that report telemetry to us, we do have a wide array of cipher suites on offer to support them all. We do support the latest and greatest protocol versions and cipher suites, so modern clients will always have the best protection available. We won't be making any changes for this item and if you'd like, you can view our SSL Labs results.

5.3 CSP configured without ‘base-uri’ directive

We had previously and consciously not set the base-uri directive in our Content Security Policy. The <base> tag in HTML allows you to set the base URL for path-relative assets, like so:

<base href="https://evil-cyber-hacker.com/">
<script src="/keylogger.js"></script>

The URL used to load the script by the browser would now be:

https://evil-cyber-hacker.com/keylogger.js

The problem the attacker would have is the JS load would still be subject to our strict script-src in our CSP and the evil-cyber-hacker.com domain is not allowed, so the script wouldn't load. Ultimately, the base-uri directive not being present wouldn't allow an attacker to do anything, but we still added it anyway!

Appendix

We then get on to some really interesting parts of the report that can only be found in the Appendix because they were not actually findings on the test, but they could have been if circumstances were different. I'm really proud of this section and it just goes to show how our Defense In Depth strategy can really pay off!

A1. CodeIgniter Validation Placeholders RCE

When Report URI was first built, it used the CodeIgniter framework exclusively. Over the years we have slowly migrated away from CodeIgniter and that migration is almost complete, with very, very little of our application even touching CodeIgniter now.

A new feature, called Validation Placeholders, that was not supported in the version of CodeIgniter we use, nor was its predecessor feature used by our code, contained a pretty serious Remote Code Execution vulnerability. You can read the full details in the security advisory, so I won't reproduce them here, but I was able to quickly evaluate the functionality of Validation Placeholders, and of our usage of Validation Rules, to determine that we were not vulnerable. Despite that, we still took robust action. I will quote directly from the report:

While Report URI used CI 3, which did not have the vulnerable feature, a patch was deployed the same day which addressed any remaining concerns. All validation rules were changed to use arrays, rather than pipe format shown in examples above. This prevents the sort of injection which enabled the placeholders to execute code. A static analysis rule was also added which forces validation rules to use Report URI’s wrapped rules, rather than CI’s default ones. Finally, the wrapped rules were changed to only accept arrays and not strings.

However, without CI 4’s placeholders, untrusted data is never injected into the execution context, as one would expect... As such, while Report URI was never exposed to this vulnerability, additional measures have been taken to further strengthen defences against it. Finally, this issue also encouraged Report URI to accelerate their transition away from CodeIgniter.

A2. Race Condition on Email Change

This was an interesting issue for the tester to come across and one that was pretty easy for us to resolve. By submitting multiple requests to change the email address on the registered account within the same HTTP/2 packet, a user account was able to be 'cloned' into several instances of the same account but with different email addresses associated.

During penetration testing, a race condition vulnerability was identified in the user email address change functionality. While the condition enabled the creation of duplicate user accounts, it was without immediate security risk, due to the robust "fail fast and fail early" principles employed by the application. The duplicates were clones, inheriting access to the original accounts' subscriptions – meaning that each duplicated account had their usage counted against the original account’s limits.

There were no security concerns presented by this issue and it only posed a problem for the person 'cloning' their account because it would break certain functionality of their account and yield no benefits. This issue is possible because of inherent limitations in Azure Table Storage not being accounted for in our code. I wrote about why I chose Table Storage all the way back in 2015.

Table Storage is a simple key:value store and it doesn't support atomic operations. Entities are inserted into a table and have two properties, the Partition Key and Row Key, that form the primary key for lookup and can't be changed. Because of the unique constraint on the combination of the Partition Key and Row Key, we store the user email address in the Row Key to prevent duplication of accounts with the same email address. The Partition Key and Row Key also can't be modified for an entity, they are the only properties that are permanent. This means to change the email address for a user we have to clone their user entity into a new entity with a different email address in the Row Key, insert the new entity and delete the old entity. It was racing the delete process that made this issue possible.

Because Table Storage doesn't support atomic operations, I couldn't fix the problem there, so I turned to our Redis cache, which does support atomic operations. By implementing a new 'user entity write lock' mechanism, we could leverage Redis to easily resolve this problem.

Here's the code to do this in Redis:

$this->redis->set('entityLock:' . $email, '1', ['NX', 'EX' => 15]);

To snip the explanation text from the issue:

The NX option will cause the write to Redis to fail if the key already exists, and the EX option sets the expiry on the key to 15 seconds. This means a user can only change their email address every 15 seconds, but fixes the race condition found in the penetration test.

To quote the penetration test report:

Despite the lack of security implications, it is worth commending Report URI’s swift remediation of the issue.

A3. Potential Issue – Path Access Control

We have some Controllers that perform functions only intended to be accessed and triggered from the Command-Line Interface. Our Router takes care of ensuring that any requests coming in from Nginx can't hit these Controllers as they are only intended to be called from the CLI. It turns out we had a small bug in our Router code and you could indeed hit these Controllers with HTTP requests. Despite this, there was no issue, and I will quote from the report:

However, all affected controllers inherited from a base command line controller class, whose constructor performed an additional verification of the execution context. Any attempt to create (access) these controllers outside of a command-line context would raise an error, as demonstrated below:

-snip-

As such, while it was never possible to reach the affected controllers, this issue highlights the importance of not making assumptions when security could be affected. Thanks to robust code and multi-layered validation, the application prevented the issue from being exploitable – and in fact, entirely mitigated the vulnerability before it was even discovered. Report URI has since patched the issue to remove any residual risk.

It's hard to imagine what someone could do with this capability, maybe they could update our aggregate count data a little more frequently, but either way, it was still a bug that needed to be fixed!

Another Success!

I think it's fair to say that this test went exceptionally well and despite all of our best efforts over the last 12 months, there are still improvements that we've had to make as a result of getting the test done. As I've said before, you really want to help a tester as much as you can to get the most value out of a test like this, and as before, here is the full, unredacted PDF report if you'd like a read.

Report URI - 2023 Penetration Test Report

I simply can't believe that Report URI has now processed 1,500,000,000,000+ reports, which is unreal! That's over one trillion, five hundred billion reports... 🤯

This tiny little project, that I had the idea of starting all those years ago, is now processing incredibly large amounts of data on a day-by-day basis. So why don't we take a look at the numbers?

Report URI

If you aren't familiar with Report URI, here's the TDLR: Modern web browsers have a heap of security and monitoring capabilities built in, and when something goes wrong on your site, they can send telemetry to let you know you have a problem. We ingest that telemetry on behalf of our customers, and extract the value from the data.

Over the years, as our customer base has grown, we are of course receiving more and more telemetry from more and more clients. This has also been expanded to include email servers which can also send telemetry about the security of emails you send! If you want any details on our product offering, the Products menu on our homepage will help you out, but this blog post isn't about that, it's about the numbers!

Our infrastructure

In a recent blog post I gave an overview of our infrastructure which really hasn't changed much over the years and has held up really well to the volume of traffic we're handling. I'll repeat the highlights here because it will help make the following data more understandable once I get into it. Here's the diagram that explains our traffic flow, along with the explanation, and you can see it's the exact same diagram I published over 5 years ago when I last spoke about it, and our infrastructure was the same long before that too:

1. Data is sent by web browsers as a POST request with a JSON payload.
2. Requests pass through our Cloudflare Worker which aggregates the JSON payloads from many requests, returning a 201 to the client.
3. Aggregated JSON payloads are dispatched to our origin 'ingestion' servers on a short time interval.
4. The ingestion servers process the reports into Redis.
5. The 'consumer' servers take batches of reports from Redis, applying advanced filters, threat intelligence, quota restrictions and per-user settings and filters, before placing them into persistent storage in Azure.

When traffic volumes are low during the day, this entire processing pipeline averages less than sixty seconds from us receiving the report to you having the data in your dashboard and visible to you. When all of America is awake and online, our busiest period of the day, we typically see processing times between three and four minutes, with the odd outliers taking possibly five or six minutes to make it through. Overall, we work as much as we can to get this time down and keep it down, but I think it's pretty reasonable.

The history of the service

I first launched Report URI in May 2015 after having worked on it and used it myself for quite some time and since then, I've covered it extensively right here on my blog. You can find the older blog posts using this tag, and the newer ones using this tag, but any change worth mentioning is something I've talked openly about. Here's a quick overview of how our report volume has grown over time.

Sep 2015 - 250,000 reports processed in a single week
Sep 2016 - 25,000,000 reports per week
Mar 2017 - 80,000,000 reports per week
Jun 2017 - 677,000,000 reports per week
Jul 2018 - 2,602,377,621 reports per week
Jun 2019 - 4,064,516,129 report per week
Mar 2021 - we hit half a trillion reports processed

That last one was a particularly big milestone and I feel like something that was really worth celebrating. Just think, half-a-trillion reports processed!

It's absolutely wild that I can now say we've processed over HALF A TRILLION REPORTS for our customers!

The current total as of this tweet stands at:

500,205,618,910 reports!!! 😲 https://t.co/jWQNQYX2dP

— Scott Helme (@Scott_Helme) March 14, 2021

We pushed on to hit our one trillionth report a little bit more quickly too! Michal even caught the point when it rolled over.

1 trillion OMG we've just processed 1 TRILLION reports 😲😍🍻 That's a loooooooooooot of JSON & XML! Good job everyone 🕺 @reporturi @Scott_Helme @troyhunt pic.twitter.com/fYohs4LmKD

— Michal Špaček (@spazef0rze) February 5, 2022

Now, as I sit here, we've pushed 1.5 trillion reports... But, enough about where we were, let's talk about where we are and what we're doing today, starting with what volumes we're processing now.

From the top!

Our first point of contact for any inbound report will be Cloudflare, so I'm going to grab the data for the last 7 days from our dashboard to take a look at. Here's just the raw number of requests that we've seen hit our edge.

In the last 7 days we saw over 3,870,000,000 requests coming in! You can see many of our common patterns trends in terms of the peaks and troughs throughout the day, and also that weekends are generally less busy than weekdays for us too. I love looking at our data egress graph because the only thing our reporting endpoint sends back is empty responses. They're usually either a 201 when we accept a report, or a 429 if you have exceeded your quota, but always empty, and there's a lot of them.

We've served a staggering 3.67TB of empty responses over the last 7 days! I also like to watch trends in how data reaches us and we can also gather some really interesting information at this scale. Take any of the following metrics for example, where we can look at how our service is being used, seeing that most people are, as expected, generating report volume in report-only mode or via our CSP Wizard.

We can also see some interesting data about clients sending reports too.

Of course, we see clients sending us many requests, but we've still received reports from over 80,000,000 unique clients!

That's a lot of browsers...

Through the Cloudflare Worker

All reports that hit our edge go through our Cloudflare Worker for processing. It does some basic sanitisation of the JSON payloads, normalisation, and maintains state about our users to know if they are exceeding their quota so reports can be dumped as early as possible. This of course means that the number of requests hitting our Worker is going to be the same as the number of requests we're receiving at the edge.

What's interesting to see is that at peak times we're receiving around 9,000 requests per second and that's a typical week for us. If a new customer joins suddenly, or an existing users deploys a misconfiguration suddenly, we've seen spikes up and over 16,000 requests per second coming in. As I mentioned in the opening paragraph though, our Worker batches up reports from multiple requests and sends them to our origin after a short period, which you can see in our Subrequests metric. Despite receiving 3,900,000,000 requests in the last 7 days, the Worker has only sent 388,310,000 requests to our origin, meaning we're batching up ~10,000 reports per request to our origin on average. This is a metric we track to fine tune our aggregation and load, but looking at it live right now, we can see that the numbers line up.

This translates to over 100 mb/s of JSON coming in to our origin per second, and bear in mind, this is massively normalised and deduplicated. Our ingestion servers take these reports and then process them into our Redis cache, so our outbound from these servers is pretty similar to the inbound, as not many reports are filtered/rejected at this stage.

Into the Redis cache

Our Redis cache for reports is a bit of a beast, as you may have guessed, and it's where reports sit for a short period of time. The cache acts as a buffer to absorb spikes and also allows for a little bit of deduplication of identical reports that arrive in a short time period, further helping us optimise. Looking at the RAM consumption, you can see reports flowing into the cache and also see when the consumer servers pull a batch of reports out for processing.

At present, we're not at our peak, but the Redis cache is handling almost 4,500 transactions per second, which isn't bad!

Consumption time!

The final stop in our infrastructure is through the consumer servers, which pull out batches of reports from the Redis cache to process into persistent storage in Azure Table Storage. We run a slightly smaller number of consumer servers with a lot more resources and these are the servers that are always being worked hard. Looking at their current CPU usage, they're sat at ~50% CPU as we start to approach the busiest time of the day, but they will still climb from here.

The consumers will only see small spikes in inbound traffic when they pull a batch of reports from Redis, but they will always have a fairly consistent outbound bandwidth to Azure as they're pushing that data out to persistent storage.

From here, it's on to the final stop for report data and after this point, it will visible in your dashboard to query, and any alerts/notifications will have been sent.

Azure Table Storage

I've used Azure Table Storage since the beginning of Report URI and it's something that I've never been given a good reason to move away from. It's a simple key:value store and is massively scalable, meaning I've never had to learn how to be a DBA and it's always taken care of for me. You can read some of my blog posts about Table Storage, but let's see how much we're using it.

This is our rate of transactions against Table Storage for the last week and as I happen to be writing this blog post on the 1st Nov 2023, all of our users have just had their monthly quota renewed. This happens at 00:00 UTC on the 1st day of each calendar month and it's why there is an enormous spike at the start of this graph and things were a lot more quiet before that. Of course, as we progress through a month, more of our users will exceed their monthly quota and reports will stop making it to persistent storage, mostly being dropped by the Cloudflare Worker and some by the consumer servers. It doesn't stop the reports being sent, it just means that we don't process them into persistent storage. Our current rate of transactions will slowly decline from where it is now down the the lowest levels by the end of the month again. As you would expect, our ingress and egress patterns follow the same trend.

Something that was introduced quite some time ago was our 90-day retention period on report data. We will keep aggregate, count and analysis data for as long as you want, but the raw JSON of each inbound report will be purged after 90 days. We had to, simply because we couldn't store that much information for an unlimited period of time. Despite that, we still have an impressive 4.9 TiB of data on disk consumed by 2,250,000,000 entities (reports)!

That's quite a lot of JSON! 😅

Where do we go from here?

Whilst all of the above are amazing numbers, you may have noticed that our current report volumes are lower than those we have previously peaked at which were detailed at the start of this post. This is a trend I've been following for a while now and I've been able to put it down to a few things. The biggest impact is that we've made really significant progress in helping our customers get up and running more quickly. Sites always send more reports and telemetry when they're first starting out using these technologies and the faster we can help them get their configurations matured, the faster they can reduce the volume of reports they're sending. Despite this, we're always adding new sites, so even though our users are using less reports on average, as we continue to grow, this has prevented our total volume from decreasing too much by constantly bringing new users on board.

Over the next year or so, we're also helping a lot of sites get ready for the new PCI DSS 4.0 requirements and we're hoping to bring larger providers on board to provide our solution through them. The PCI SSC are putting a huge pressure on sites with an e-commerce component to protect against Digital Skimming Attacks (a.k.a. Magecart) by locking down the JavaScript on their payment pages, something that CSP was literally designed for! As the natural choice for a reporting platform, we're well suited to help sites get their CSP defined, tested and deployed with the least friction possible.

I'm excited for what the rest of 2023 will bring, but I'm looking forward to 2024 already. Having built this company up from the first line of code almost 9 years ago, to where we are today, I wonder if it might be time to take the next big step soon! 😎

The Security Headers grading criteria is something that doesn't change often, but when it does, there's a good reason behind the change. In this blog, I will outline the new grading criteria and the reasons why we've made the change.

Information: This is a cross-post from the Probely blog, you can read the original article there.

Security Headers

For those that aren't familiar, Security Headers is a free security scanning tool that takes only a couple of seconds to conduct a scan. Your site can be awarded anything from an A+ grade, down to an F grade, all depending on the configuration of your HTTP Response Headers.

If you want to conduct a scan, head over there now to check your grade and you will be tested against the new criteria outlined below.

Existing Criteria

The grading criteria is quite clear as all of the headers that we score are listed in the summary section, making it easy to see which headers you're getting right, and which headers you're missing. Take for example the scan of my personal blog here:

You can see that I have all of the headers present, which is why they're highlighted in green, but I'm still not achieving the highest possible grade of an A+ because I have a warning.

This warning is the most frequently raised question about our grading, and in truth, it's basically the only thing that's ever raised with our grading.

Using 'unsafe-inline' in CSP style-src

The warning here is given because I have the 'unsafe-inline' value in my CSP style-src directive, allowing me to use inline styles on the page. Sites will sometimes do this because it's convenient, because they use a theme that requires it, because one of the libraries they use requires it, or a whole bunch of other reasons. An inline style might look like one of the following:

<html>
  <head>
    <style>
      p {
        color: green;
      }
    </style>
  </head>
</html>

<p style="color: green;">This paragraph is green.</p>

The problem with allowing unsafe inline styles is that an attacker can inject styles into the page and the browser wouldn't know if they were put there by the host of the website or by an attacker, meaning they'd be used on the page.

As a good example of where 'unsafe-inline' might be required, just take a look at the CSP docs for Angular:

The minimal policy required for brand new Angular is:
default-src 'self'; style-src 'self' 'unsafe-inline';

Can you get pwned with CSS?

It's an interesting question, and one I've been considering for a long time now. I even put the question out to the wider community back in Feb 2022 with a Twitter poll and a blog post by the same title: Can you get pwned with CSS?

Allowing 'unsafe-inline' in the script-src directive of CSP is obviously dangerous and can lead to an almost infinite number of bad things happening, but as it turns out, the damage that can be done with styles is quite limited. Not only that, but I've yet to find any credible evidence of an attack happening where malicious styles were the attack vector. Even with widespread feedback from the community, the hypothetical issues are few and far between.

What are we trying to achieve?

For me, any question on our grading criteria comes back to this point. Which of the following are we aiming for?

1. The absolute best configuration possible, regardless of anything.
2. A sensible balance that it would be beneficial for everyone to achieve.

As time has gone by, I've seen security absolutists push us towards #1 on more occasions than I can count, but also as time goes by, I've realised that achieving #2 would take us to a far better place overall. This is what we're focusing on with this change.

Allowing 'unsafe-inline' in CSP style-src

As of the publication of this blog post, Security Headers will be relaxing the grade cap penalty for using 'unsafe-inline' in CSP style-src which will, all else being good, allow you to achieve an A+ grade! The warning will still be present, because long term we should still try to move away from it, or at least not introduce it, but the grade cap will no longer ding your score.

To be clear, a site will still have had to put serious thought and effort into the configuration of their Security Headers, but now, I think the A+ grade means you've achieved something that I'd be happy if all sites were to achieve.

Scan your site for free and tag us in your results! You'll get a cool badge when sharing a link to your scan results on social media! 😎

Almost 2 years on from the last time I wrote about QWACs, I'm sadly not here to tell you that things have gone well since then. In fact, I'm actually here to tell you that things are not going well at all...

QWAC

Back in Jan 2022, I wrote a blog post that went into details on what a QWAC, or Qualified Website Authentication Certificate, actually is: If it looks like a duck, swims like a duck, and QWACs like a duck, then it's probably an EV Certificate

TLDR; It's an EV Certificate all over again 🤷‍♂️

In all seriousness though, that's actually quite a long and detailed post about the shortcomings of a QWAC and why they're just a terrible, terrible idea. They're only being pushed by organisations that would make $$$ selling them (funny that) and it's like the entire mess of EV has been conveniently forgotten. I'm not here to re-tread the same ground, though, I'm here to talk about something even more concerning. You might think "ok, so we have a new type of pointless certificate available", and if that were the case, I wouldn't be writing about it again and we could all just not buy them. The problem is that there's something bigger lurking that really concerns me.

My Concerns

This isn't all just talk for me, having dedicated a huge portion of my life to working in this industry and being so passionate about it, this worries me. It worries me enough that I've signed multiple open letters speaking out against this, with the most recent just a few days ago, and I've even travelled to Brussels to sit alongside Member of European Parliament Karen Melchior, and other industry representatives, to speak against this. I have absolutely no skin in this game, one way or another, but I've seen something that I believe is just fundamentally wrong, and I feel compelled to speak out against it.

eIDAS Article 45 - latest recitals

As we come towards the end of the legal process, we're closing in on the final revisions and final draft of some new regulation coming to the EU called eIDAS. This new regulation contains many things, and it's only one small part of it that I fundamentally oppose, but it will have Global impact, far beyond the borders of any member state of the EU.

Alongside introducing the concept of a QWAC, discussed in my previous blog post, eIDAS is also going to introduce some very concerning requirements that affect the Internet PKI. At the top of my list of concerns is that browser and client vendors (Root Store Operators) will have a legal obligation to add Government mandated Root Certificate Authorities to their Root Stores, bypassing existing approval mechanisms.

Yep, you read that right. Government mandated Root Certificate Authorities...

I could end this blog post right here because anyone reading this will understand the significance of such a statement, and just how much of a catastrophically bad idea that is, but it gets worse. There will also be restrictions placed on Root Store Operators around handling incidents at those Root CAs and possibly removing trust in them for wrongdoing. I cannot stress this enough so I'm going to say it again, this is a terrible idea.

How it works now

The system that we have now is not perfect, by any stretch of the imagination, but it has been improved so much over the years with tireless work from the industry, that where we are now, finally, is a good place.

A browser or device vendor like Apple has a collection of Trusted Root Certificate Authorities that their devices will trust, and in turn, those devices will trust any certificates issued by those Trusted Root CAs. If you want to join this collection of Trusted Root CAs, you have to apply to join the Apple Root Certificate Program and pass some very strict requirements. Of course, this makes sense, because being a Trusted Root CA is a massive responsibility that gives you an enormous amount of power, and Apple want to make sure that their customers aren't going to come to any harm because of your actions. The same goes for all such Root Store Operators like Mozilla, Chrome, Microsoft and many others that operate Trusted Root Programs for their own devices or software. It is in the interest of the software/device vendor to make sure that a Root CA is capable of operating properly because if not, all of that vendor's customers are at serious risk of having their traffic intercepted and decrypted. So, for Apple, their concern is that if a Root CA makes a mistake, the potential outcome is that everyone using an iPhone could have the security of all of their traffic compromised! That's a serious risk, and it's why organisations like Apple take the process of approving Trusted Root CAs so damn seriously.

This is the existing approval mechanism that will be bypassed by this new legislation and the Root Store Operators will be required to accept these European Root CAs without the ability to scrutinise them, or, reject their inclusion.

How it's going to work

I have access to the near-final text of the regulation, which is not yet public, but was leaked to me by a confidential source. I've been looking through the proposed changes and I still see all of the things that have concerned me throughout this entire process. Here are a few snippets from the hundreds of pages that I've read through that still demonstrate my concerns. These snippets outline the definition of a QWAC and that they must be held against the standards set out in the legislation:

‘qualified certificate for website authentication’ means a certificate for website authentication, which is issued by a qualified trust service provider and meets the requirements laid down in Annex IV;

Qualified certificates for website authentication shall meet the requirements laid down in Annex IV.

Evaluation of compliance with those requirements shall be carried out in accordance with the standards and the specifications referred to in paragraph 3.

But if that isn't clear enough for you, the legislation goes on to say:

Qualified certificates for website authentication issued in accordance with paragraph 1 shall be recognised by web-browsers. Web-browsers shall ensure that the identity data attested in the certificate and additional attested attributes are displayed in a user-friendly manner. Web-browsers shall ensure support and interoperability with qualified certificates for website authentication referred to in paragraph 1

That's pretty clear, and we can still see the same concerns I raised previously about the legislation controlling not only support for, and use of, the Government Mandated Root CAs, but even control over the UI of the browser itself. It goes on:

National trusted lists should confirm the qualified status of website authentication services and of their trust service providers, including their full compliance with the requirements of this Regulation with regards to the issuance of qualified certificates for website authentication. Recognition of QWACs means that the providers of web-browsers should not deny the authenticity of qualified certificates for website authentication attesting the link between the website domain name and the natural or legal person to whom the certificate is issued and confirming the identity of that person. Providers of web-browsers should display in a user-friendly manner the certified identity data and the other attested attributes to the end-user, in the browser environment, by relying on technical implementations of their choice. To that end, providers of web-browsers should ensure support and interoperability with qualified certificates for website authentication issued in full compliance with the requirement of this Regulation.

Again, pressing this idea of a list of Trusted Root CAs that the client vendors must accept and "should not deny the authenticity of". Then, with regards to limiting the ability of a Root Store Operator to audit the behaviour of a Trusted Root CA on an ongoing basis:

In order to contribute to the online security of end-users, providers of web-browsers should be able to take measures, in exceptional circumstances, that are both necessary and proportionate in reaction to substantiated concerns on breaches of security or loss of integrity of an identified certificate or set of certificates. In this case, while taking any such precautionary measures, web browsers should notify without undue delay the national supervisory body and the Commission, the entity to whom the certificate was issued and the qualified trust service provider that issued that certificate or set of certificates of any such concern of a security breach as well as the measures taken relating to a single certificate or a set of certificates. These measures, should be without prejudice to the obligation of the browsers to recognize qualified website authentication certificates in accordance with the national trusted lists.

Then, just to make sure we don't have any tremendously beneficial technologies like Certificate Transparency protecting us, it is clarified that:

Qualified certificates for website authentication shall not be subject to any mandatory requirements other than the requirements laid down in paragraph 1.

Paragraph 1, of course, does not make any mention of Certificate Transparency. All of these points are then summarised in a newly added section with the title "Cybersecurity precautionary measures":

1. Web-browsers shall not take any measures contrary to their obligations set out in Art 45, notably the requirement to recognise Qualified Certificates for Web Authentication, and to display the identity data provided in a user friendly manner.

2. By way of derogation to paragraph 1 and only in case of substantiated concerns related to breaches of security or loss of integrity of an identified certificate or set of certificates, web-browsers may take precautionary measures in relation to that certificate or set of certificates

3. Where measures are taken, web-browsers shall notify their concerns in writing without undue delay, jointly with a description of the measures taken to mitigate those concerns, to the Commission, the competent supervisory authority, the entity to whom the certificate was issued and to the qualified trust service provider that issued that certificate or set of certificates. Upon receipt of such a notification, the competent supervisory authority shall issue an acknowledgement of receipt to the web-browser in question.

4. The competent supervisory authority shall consider the issues raised in the notification in accordance with Article 17(3)(c). When the outcome of that investigation does not result in the withdrawal of the qualified status of the certificate(s), the supervisory authority shall inform the web-browser accordingly and request it to put an end to the precautionary measures referred to in paragraph 2.

The industry speaks out

It's not just me that thinks this is a bad idea though, of course, I'm just adding my voice to the chorus of other voices from across industry.

1. Mozilla set up the Security Risk Ahead website with lots of details.
2. The Chrome Security Team has called for change in Qualified certificates with qualified risks.
3. You can head over to https://last-chance-for-eidas.org/ to read more about the risks.
4. You can read our latest open letter with 400+ signatures so far. https://eidas-open-letter.org/

The thing that it will always come down to for me, and the thing that you can use to guide your decisions, is to look at the interests of the parties involved. I've long been critical of many CAs for shitty marketing and shady practises, and it seems that's continuing. The organisations and voices speaking out in support of QWACs and Article 45 are those that are going to be able to sell them for $$$ once this comes to pass. The organisations and voices speaking out against QWACs and Article 45 are those with an interest in preserving and improving the security of the ecosystem we've worked so hard to build. I have nothing to gain here by swaying your opinion, but you sure as hell have a lot to lose.

What do we do about it?

I'll quote the following snippet from the 'Last Chance' website:

If you’re a European citizen, you can write to the member of the European Parliament responsible for the eIDAS file - Romana JERKOVIĆ - and register your concern.

If you’re a cybersecurity expert, researcher or represent an NGO, consider signing the open letter at https://eidas-open-letter.org.

In truth, I don't know what else to do next, but I believe we have to do something. If these Qualified Trust Service Providers (QTSP is the name given to a CA that issues QWACs) are all they're cracked up to be, then why can't they just submit to the existing audit/approval process and pass with flying colours?.. That's not too much to ask, is it?

Additional information and reading

Timeline of Certificate Authority Failures - why Trust Store Operators need the ability to audit and remove Root CAs.

Mozilla website pushes serious eIDAS misinformation to political decision makers and public - The ESD (a group of CAs) produced this laughable document. It closes by pointing out that Google and Mozilla are "investors" in Let's Encrypt who are "in competition with all QTSPs" 😂 (a QTSP is a CA that issues QWACs)

Digital rights organisation epicenter.works had this to say about QWACs.

You should read what Alec Muffett has to say on eIDAS/QWACs.

This informative Tweet from Ryan Hurst is also a great start for info on the Internet PKI.

Update 19:10 UTC 7th Nov: The EFF have just published something on this, Article 45 Will Roll Back Web Security by 12 Years, and as you would expect, it's well written and makes a lot of sense!

I love having smart devices around my house, and every now and then, you can have a little bit of fun with them too! Here's what it currently looks like from the outside of my house for Halloween.

Custom doorbell sounds

As Halloween has arrived, I thought it might nice to style the outside of my house using all of the RGB exterior lighting I've installed, and to play a custom sound from the doorbell when visitors press the button. It's pretty quick to do this and you might spend most of the time finding a suitable audio file to play when the button is pressed!

Enable SSH on your UniFi Console

The first thing you need to do is setup SSH access on your UniFi Console. This could be a UDM Pro, a CloudKey, or, like me, a UniFi NVR device. You can find specific instructions on how to do this here: https://help.ui.com/hc/en-us/articles/204909374-UniFi-Connect-with-SSH-Advanced

Once you have SSH access to your console, connect using the default credentials, or any specific ones you set, and modify the following file to enable SSH access on your devices like cameras and doorbells:

vi /srv/unifi-protect/config.json

Update Nov 2023: The file path changed.
vi /etc/unifi-protect/config.json

If the file doesn't exist, you can create it and add the following content:

{
 "enableSsh": true
}

If the file does exist, you can edit it and add the required config, making sure to keep the syntax in the JSON file valid.

{
 "someExistingKey": "someExistingValue",
 "enableSsh": true
}

Now restart the UniFi Protect service to load the new configuration.

systemctl restart unifi-protect

Connect to the Doorbell via SSH

You can now test the connection to your doorbell for which you will need to know the IP address, which can be found in the 'UniFi Devices' list in Protect. Here is the SSH command to connect, just substitute my IP address with yours:

ssh ubnt@192.168.1.140

You will be asked for a password and the password for the device can be found in the Protect dashboard by going to Settings -> System and clicking 'Reveal' under the 'Recovery Code' heading.

I was automatically dropped into the correct folder when I connected to the doorbell, but just to make sure, you can use this command to make sure you're in the right place:

cd /var/etc/persistent

Set up your custom chime sound!

You can now edit the configuration file in this folder to make the changes on the doorbell that we need to make for our custom sound:

vi ubnt_sounds_leds.conf

The path we need to change is sounds_ring_button and I also took the opportunity to change speakerVolume to 100 while I was here!

{
  "customSounds": {
    "sounds_ring_button": "../../../../../../var/etc/sounds/custom.wav"
  },
  ...
  "speakerVolume": 100,
  ...
}

The final piece of the puzzle is to copy over your WAV file to the doorbell and put it in the right folder:

/var/etc/sounds/custom.wav

You will need to update the filename if you don't call it custom.wav and you can copy it over using an scp command or a tool like WinSCP if you're more comfortable with that:

scp custom.wav ubnt@192.168.1.140:/var/etc/sounds/custom.wav

Now all that's left is to restart the correct process by finding it's pid, and you're good to go. Here is how you find the pid:

ps | grep ubnt_sounds_leds

It will give you output similar to this, and the pid for ubnt_sounds_leds is the first number on the line, so for me it was 2560 but it will be different for you:

 2560 ui        159m S    /bin/ubnt_sounds_leds
 6710 ui        2968 S    grep leds

You can then kill the pid, which will be restarted for you automatically, by using this command and replacing my pid with your pid:

kill -TERM 2560

That's it, go and hit the button!!

I'm planning to utilise this again for Christmas and perhaps just for a little more fun throughout the year too. My 10 year old also wants to know if the doorbell can play fart sounds... 😂

I'm thoroughly pleased to be able to say that I finally understand the issue that's been bothering me on Report URI for a few weeks now, and this is the blog post that's going to explain everything!

Report URI

If you haven't read the first two blog posts related to this issue, it might help you to get some background on what I'm going to be talking about here:

Unravelling The Mystery Of Truncated POST Requests On Report URI

Processing Truncated Requests? A PHP Debugging Deep Dive

This weird behaviour of our application receiving and then trying to process truncated POST requests has lead me on quite a journey of learning, and I can finally explain what's happening!

The Story So Far

In the previous blog posts, I'd narrowed it down to what was happening, but not why it was happening, and I was still stumped. After patching PHP, I could see it was calling read() on the socket and reading until 0 was returned, but 0 was being returned before expected. Here's the log of the behaviour from the previous blog post.

Sep 29 20:23:31 test-server php: Accepted connection: 0ce264eae07198c3c59ae90d04127c39
Sep 29 20:23:31 test-server php: Read 16384 bytes (1024737 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (1008353 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (991969 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (975585 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (959201 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (942817 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (926433 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (910049 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (893665 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (877281 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (860897 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (844513 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (828129 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (811745 bytes remaining)
Sep 29 20:23:31 test-server php: errno 0: Success
Sep 29 20:23:31 test-server php: Read 0 bytes (811745 bytes remaining)
Sep 29 20:23:31 test-server php: POST data reading complete. Size: 229376 bytes (out of 1041121 bytes)

I knew that Nginx was receiving the full request, so the problem was happening in the fastcgi_pass to PHP-FPM, but I couldn't wrap my head around the circumstances. I wanted to understand more on the underlying mechanics of how Nginx was talking to PHP and, more specifically, what was failing in that process.

bpftrace

In the last blog I had to get familiar with strace for the first time, and this time around, I'd asked for some assistance from Ignat Korchagin who kindly whipped up several bpftrace commands, and, could also reliably reproduce the issue. Hopefully, by monitoring things a little more closely between Nginx and PHP, I could gain an understanding of precisely what was happening.

After monitoring my test server when the issue was happening, I could see Nginx hitting EAGAIN (source) when trying to write to the socket, and I could see PHP getting EPIPE (source) when trying to read from the socket, but it still wasn't clear to me how that was occurring. The EAGAIN error is likely because the socket buffer is full when Nginx is sending, so it can't send any more data, and the EPIPE comes when PHP tries to read from the socket that has since been closed by Nginx. Here's the command.

$ bpftrace -e 'kretfunc:unix_stream_sendmsg /retval < 0/ { printf("%d\n", retval); }'
Attaching 1 probe...
-11
-11
[-11 repeated 20x]
-11
-11
-32
-32
[-32 repeated 12x]
-32
-32

The log reads like Nginx is trying to write to the socket which has a full buffer, and PHP is trying to read from a socket that's closed, but if PHP has called accept(), why would it not be able to read()? The accept() call is followed by read() calls in the PHP source almost immediately so it didn't make sense. I also saw 12x EPIPE (-32) errors coming from PHP and I had exactly 12x truncated payload errors in syslog using the same logging from the previous blog post.

Digging a little deeper, I wanted to see when the calls to connect() and accept() were being made.

$ bpftrace -e 'tracepoint:syscalls:sys_enter_connect /comm == "nginx"/ { printf("connect: %s\n", strftime("%H:%M:%S:%f", nsecs)); } tracepoint:syscalls:sys_enter_accept /comm == "php-fpm8.2"/ { printf("accept: %s\n", strftime("%H:%M:%S:%f", nsecs)); }'

After firing another batch of 20 requests, I could see all of the connect() calls from nginx happen right away, but the accept() calls from PHP were much more spaced out.

Attaching 2 probes...
connect: 10:42:27:244851
connect: 10:42:27:576388
connect: 10:42:27:643525
connect: 10:42:27:750282
connect: 10:42:27:923155
connect: 10:42:28:054478
connect: 10:42:28:419101
accept: 10:42:28:522173
connect: 10:42:28:716231
connect: 10:42:29:129984
connect: 10:42:29:146868
connect: 10:42:29:240262
connect: 10:42:29:307529
connect: 10:42:29:310247
accept: 10:42:29:523505
connect: 10:42:29:825400
connect: 10:42:29:832304
connect: 10:42:29:978008
connect: 10:42:30:155472
connect: 10:42:33:271992
connect: 10:42:35:891986
connect: 10:42:37:164266
accept: 10:42:37:257787
accept: 10:42:37:588539
accept: 10:42:38:534605
accept: 10:42:39:534456
accept: 10:42:47:260393
accept: 10:42:47:590909
accept: 10:42:48:537622
accept: 10:42:49:536993
accept: 10:42:57:263766
accept: 10:42:57:593434
accept: 10:42:58:540436
accept: 10:42:59:539849
accept: 10:43:07:267292
accept: 10:43:07:595731
accept: 10:43:08:543073
accept: 10:43:09:542913
accept: 10:43:17:271296
accept: 10:43:17:598086

I still wasn't clear on how this was working because PHP is calling accept() and read() very close together, so if PHP can accept, why is the read failing with a partial payload?!

Oct 12 10:42:47 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:42:47 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:42:48 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:42:49 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:42:57 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:42:57 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:42:58 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:42:59 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:43:07 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:43:07 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:43:08 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:43:09 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:43:17 test-server php: JSON decode failed: 1041121 229376
Oct 12 10:43:18 test-server php: message repeated 2 times: [ JSON decode failed: 1041121 229376]
Oct 12 10:43:19 test-server php: JSON decode failed: 1041121 229376

The truncated payload issued showing in syslog matched up to the late accept() calls from PHP! It was at this point I was prompted to look at the backlog parameter of listen() and it became clear that this behaviour can be explained by backlog, something I wasn't familiar with.

int listen(int sockfd, int backlog);

When calling listen(), the second parameter provided is backlog, which is described as follows in the docs:

The backlog argument defines the maximum length to which the
queue of pending connections for sockfd may grow.

So now we can have an application, like Nginx, call connect() and succeed but without the application on the other side, PHP, calling accept()! This is finally starting to make a little more sense with the behaviour I'm seeing. It also turns out that Nginx can write() until the receive buffer on the socket is full, and that is when it's going to hit EAGAIN. I confirmed this using strace:

cat nginx.strace | grep -E 'sendfile\(|connect\(|close\(|writev\('
75920 19:43:26 connect(3, {sa_family=AF_UNIX, sun_path="/run/php/php8.2-fpm.sock"}, 110) = 0
75920 19:43:26 writev(3, [{iov_base="\1\1\0\1\0\10\0\0\0\1\0\0\0\0\0\0\1\4\0\1\2[\5\0\t\0PATH_I"..., iov_len=648}], 1) = 648
75920 19:43:26 sendfile(3, 29, [0] => [32768], 32768) = 32768
75920 19:43:26 writev(3, [{iov_base="\1\5\0\1\200\0\0\0", iov_len=8}], 1) = 8
75920 19:43:26 sendfile(3, 29, [32768] => [65536], 32768) = 32768
75920 19:43:26 writev(3, [{iov_base="\1\5\0\1\200\0\0\0", iov_len=8}], 1) = 8
75920 19:43:26 sendfile(3, 29, [65536] => [98304], 32768) = 32768
75920 19:43:26 writev(3, [{iov_base="\1\5\0\1\200\0\0\0", iov_len=8}], 1) = 8
75920 19:43:26 sendfile(3, 29, [98304] => [131072], 32768) = 32768
75920 19:43:26 writev(3, [{iov_base="\1\5\0\1\200\0\0\0", iov_len=8}], 1) = 8
75920 19:43:26 sendfile(3, 29, [131072] => [163840], 32768) = 32768
75920 19:43:26 writev(3, [{iov_base="\1\5\0\1\200\0\0\0", iov_len=8}], 1) = 8
75920 19:43:26 sendfile(3, 29, [163840] => [196608], 32768) = 32768
75920 19:43:26 writev(3, [{iov_base="\1\5\0\1\200\0\0\0", iov_len=8}], 1) = 8
75920 19:43:26 sendfile(3, 29, [196608] => [229376], 32768) = 32768
75920 19:43:26 writev(3, [{iov_base="\1\5\0\1\200\0\0\0", iov_len=8}], 1) = -1 EAGAIN (Resource temporarily unavailable)

I believe that Nginx is using writev() to send the fastcgi headers, and using sendfile() to send chunks of the body, which comes from this function in the code. It certainly seems like this is starting to make a little sense. Looking at the PHP config, I wanted to see what the backlog value was because it's not something I ever recall setting. I found the following in our config:

; Set listen(2) backlog.
; Default Value: 511 (-1 on Linux, FreeBSD and OpenBSD)
;listen.backlog = 511

With a default value of -1, I was guessing that backlog might be cast to an unsigned int by the kernel, which it is (source).

if ((unsigned int)backlog > somaxconn)
	backlog = somaxconn;

You can also see that, as stated in the manual (source), if the defined backlog value is greater than somaxconn, it will be silently truncated to that value. Because -1 will actually be 4294967295, by default, PHP will always set backlog to the value of somaxconn, effectively letting the system decide.

# cat /proc/sys/net/core/somaxconn
4096

This means we can have up to 4,096 connections pending in the backlog queue, which is quite a few!

The Theory

At this point, I believe something like the following is happening during the lifecycle of the request.

1. Nginx makes the connect() syscall and begins writing to the socket.
2. While PHP is able, it processes the initial requests. We see a small number of requests succeeding.
3. PHP becomes overloaded and stops processing new, inbound requests.
4. Nginx continues to make new connections and fill the socket buffer with data.
5. Once the receive buffer is full, Nginx will receive EAGAIN and likely wait for something like fastcgi_send_timeout (source).
6. The send will timeout and Nginx will close the connection.
7. PHP becomes available and accepts the connection, starting to read until the receive buffer has been consumed.
8. Once the buffer has been read, PHP will get EPIPE and process the content read from the buffer.

Sounds logical, right? It should also be easy enough to test this now that I know what I'm aiming for so I tinkered with some of the config parameters in Nginx and it did indeed seem fastcgi_send_timeout was the main factor. By reducing the send timeout, I would see more requests failing to parse in general, and I assume that's because Nginx is timing out sooner so PHP has less time to get to the backlog.

Testing The Theory!

Now for the fun part, where you actually find out if you were right, or you were completely off the mark! I was looking into ways that I could reliably reproduce this and my first effort was to take a few steps. I reduced the number of child processes PHP can spawn to one, so it would be easy to keep it busy, and I reduced the send timeout on Nginx to three seconds so it would time out very quickly. With this basic setup I could send two requests sequentially and the first one would succeed and the second one would reliably fail, 100% of the time. I wanted something better than that though, something that would allow me to trigger this on a single request, and that's when João Poupino dropped this little gem in Slack!

This gives me the ability to talk directly to the PHP socket, using the FastCGI protocol, and completely cut Nginx out of the equation. The script also has a feature that allows you to send only a partial request, where it will randomly close() the socket somewhere during the transmission of the payload. With this, it should now be possible to reproduce this problem with only a single request. The moment of truth....

python3 send-fastcgi-request.py unix:/run/php/php8.2-fpm.sock post.data --partial
[102935] Sending request 1/1
[102936] Connecting to unix:/run/php/php8.2-fpm.sock...
[102936] Sending 37 FastCGI records...
[102936]   * Sent record 1/37 (8 + 8 bytes)
[102936]   * Sent record 2/37 (8 + 512 bytes)
[102936]   * Sent record 3/37 (8 + 0 bytes)
[102936]   * Sent record 4/37 (8 + 32768 bytes)
[102936]   * Sent record 5/37 (8 + 32768 bytes)
[102936]   * Sent record 6/37 (8 + 32768 bytes)
[102936]   * Sent record 7/37 (8 + 32768 bytes)
[102936]   * Sent record 8/37 (8 + 32768 bytes)
[102936]   * Sent record 9/37 (8 + 32768 bytes)
[102936]   * Sent record 10/37 (8 + 32768 bytes)
[102936]   * Sent record 11/37 (8 + 32768 bytes)
[102936]   * Sent record 12/37 (8 + 32768 bytes)
[102936]   * Sent record 13/37 (8 + 32768 bytes)
[102936]   * Sent record 14/37 (8 + 32768 bytes)
[102936]   * Sent record 15/37 (8 + 32768 bytes)
[102936]   * Sent record 16/37 (8 + 32768 bytes)
[102936]   * Sent record 17/37 (8 + 32768 bytes)
[102936]   * Sent record 18/37 (8 + 32768 bytes)
[102936]   * Sent record 19/37 (8 + 32768 bytes)
[102936]   * Sent record 20/37 (8 + 32768 bytes)
[102936]   * Sent record 21/37 (8 + 32768 bytes)
[102936]   * Sent record 22/37 (8 + 32768 bytes)
[102936]   * Sent record 23/37 (8 + 32768 bytes)
[102936]   * Sent record 24/37 (8 + 32768 bytes)
[102936] Closing socket after record 24!

The script started to send my large POST payload and decided to close the socket after sending only 24 out of the 37 chunks of the request. Time to check syslog and see if we can find what I'm hoping for and expecting.

Oct 13 13:33:57 test-server php: POST data reading complete. Size: 688128 bytes (out of 1069923 bytes)
Oct 13 13:34:07 test-server php: JSON decode failed: 1069923 688128

Finally! 🍾🥂

It feels awesome to get to the bottom of something that's been bugging you so much! I fixed the issue a whole two blog posts ago, and I could have moved on, but sometimes, I really just need to understand why something is happening, and honestly, it was quite a fun journey getting here. I owe a big thanks to João and Ignat, who collectively must have saved me countless days of effort that I might not even have had time for.

I've also skipped over a few of the debugging steps above because this was becoming another lengthy blog post, but I did, during and after debugging, spend a lot of time trawling through the Nginx source to understand fully what's happening and also just to confirm my understanding.

The Nginx behaviour of using writev() and sendfile() can be seen here: https://github.com/nginx/nginx/blob/9f8d60081cd4eefa5fcf0df275d784d621290b9b/src/os/unix/ngx_linux_sendfile_chain.c#L50

Nginx adds a timeout when receiving EAGAIN here, using fastcgi_send_timeout. I also noticed in the Nginx source that it makes use of epoll(), which explains earlier why I was only seeing a single EAGAIN per request: https://github.com/nginx/nginx/blob/9f8d60081cd4eefa5fcf0df275d784d621290b9b/src/http/ngx_http_upstream.c#L2093

The event timer expires in Nginx here: https://github.com/nginx/nginx/blob/9f8d60081cd4eefa5fcf0df275d784d621290b9b/src/event/ngx_event_timer.c#L54

I also spotted that PHP was respawning the child processes when I was testing and that they'd always have new PIDs after a test run. It seems that if the child is idle for too long, PHP will kill and respawn it, possibly based on this timeout value: https://github.com/php/php-src/blob/47c6b3bd452d2932af5c2f021b10ab2aaed01fb1/sapi/fpm/fpm/fpm_conf.c#L622

All in all, I can now say I'm happy that I understand everything that was happening with this issue and can finally put it to bed!

As a final closing note, just to nicely close things off, I got a response on the PHP bug that I filed in the previous blog post and it seems that this finding is indeed a bug and will be patched!

In my previous blog post, I came across a bug in Report URI that took some effort to debug and fully understand before I could fix it. Whilst I'd identified what the issue was, and how it was happening, I never got to the bottom of why it was happening. In this post, I'm going to delve into the PHP source to figure it out!

Truncated POST Requests

If you haven't read my previous blog post, Unravelling The Mystery Of Truncated POST Requests On Report URI, you should start there as it sets the scene for what I'm investigating here. It's a bit lengthy, but it has all of the details that are required to understand the journey in this post. Just like in that previous post, I'm investigating things outside of my area of knowledge, but sometimes, I have an itch that I just have to scratch, and this was one of those times!

It Wasn't Nginx

At the end of the last blog post, I wasn't 100% sure whether Nginx was at fault, or whether PHP was at fault. Admittedly, I did have a feeling that it was going to be PHP that was responsible, but I hadn't tested my idea and didn't have any evidence, so I didn't share that view at the time.

The problem we had was that POST requests were coming through and by the time they got to PHP the payload was truncated, and that caused a failure in parsing the JSON payload because it was invalid. I knew that Nginx was receiving the full payload and we use the default value for fastcgi_request_buffering (link) so "the entire request body is read from the client before sending the request to a FastCGI server". I could verify this with some simple debug logging in Nginx, which I did, and the failure was happening between Nginx and PHP during the fastcgi_pass. Nginx has the full request body to pass, it's just not making it to the other side because the socket is closed before everything can be sent.

Previously, we were using a TCP socket between Nginx and we were experiencing a lot of memory pressure on TCP resources. That resulted in an out of memory - consider tuning tcp_mem error (source) and Linux does warn us that it will "kill" sockets if there is "strong memory pressure" (source).

bool tcp_check_oom(struct sock *sk, int shift)
{
	bool too_many_orphans, out_of_socket_memory;

	too_many_orphans = tcp_too_many_orphans(shift);
	out_of_socket_memory = tcp_out_of_memory(sk);

	if (too_many_orphans)
		net_info_ratelimited("too many orphaned sockets\n");
	if (out_of_socket_memory)
		net_info_ratelimited("out of memory -- consider tuning tcp_mem\n");
	return too_many_orphans || out_of_socket_memory;
}

/**
 *  tcp_out_of_resources() - Close socket if out of resources
 *  @sk:        pointer to current socket
 *  @do_reset:  send a last packet with reset flag
 *
 *  ...
 *
 *  Criteria is still not confirmed experimentally and may change.
 *  We kill the socket, if:
 *  1. If number of orphaned sockets exceeds an administratively configured
 *     limit.
 *  2. If we have strong memory pressure.
 *  3. If our net namespace is exiting.
 */
static int tcp_out_of_resources(struct sock *sk, bool do_reset)
{

Since we swapped away from a TCP socket to a Unix socket, the same problem could still be reproduced, but instead by pressuring PHP.

Reproducing the error with a Unix Socket

Using the same slow script from the previous blog, with a small tweak to remove the outbound TCP connection, we can keep all of the PHP processes busy easily.

sleep(30);

$json = json_decode(file_get_contents('php://input'));

if ($json == null) {
  syslog(LOG_WARNING, "JSON decode failed: " . $_SERVER['CONTENT_LENGTH'] . " " . strlen(file_get_contents('php://input')));
}

I then tweaked the Nginx config so that it would timeout on fastcgi operations quickly.

fastcgi_connect_timeout 3s;
fastcgi_send_timeout 3s;

The new Nginx config, coupled with the really long running PHP scripts, means that it will be easy to get Nginx to start timing out on the fastcgi_pass and that's when the error occurs.

Debugging PHP

Fortunately, I had quite a specific area of the php-src I'd need to look at to investigate so this would help me narrow down quite quickly. We're running v8.2.10 in production on Report URI and I've also verified my findings here against the latest 8.2 branch which is v8.2.11 at the time of writing. I will do my work here against that branch.

Following the processing of a request, we go through:

php_request_startup()
sapi_activate()
sapi_read_post_data()
SAPI_POST_READER_FUNC()
sapi_read_post_block()
sapi_module.read_post()
sapi_fcgi_read_post()
fcgi_read()
safe_read()

In safe_read(), we make the final system call to read(), to read data from the socket. I've snipped the WIN32 code for ease of reading because it isn't relevant here.

static inline ssize_t safe_read(fcgi_request *req, const void *buf, size_t count)
{
	int    ret;
	size_t n = 0;

	do {
#ifdef _WIN32
		/* snip */
#endif
		errno = 0;
#ifdef _WIN32
		/* snip */
#else
		ret = read(req->fd, ((char*)buf)+n, count-n);
#endif
		if (ret > 0) {
			n += ret;
		} else if (ret == 0 && errno == 0) {
			return n;
		} else if (ret <= 0 && errno != 0 && errno != EINTR) {
			return ret;
		}
	} while (n != count);
	return n;
}

Looking at the manual page for read(), I notice a couple of things that stand out in Description and later in Return Value:

If the file offset is at or past the end of file, no bytes are read, and read() returns zero.

On success, the number of bytes read is returned (zero indicates end of file)

The PHP source has handling for ret being zero, and also checks errno in the two else if statements, but I wasn't sure what was being returned by read(). I decided to add some of my own logging to see exactly what was going on by patching PHP. I spun up a test server with some hefty resources so I'd be able to build PHP quickly and set it up as follows.

add-apt-repository ppa:ondrej/php
add-apt-repository ppa:ondrej/nginx
apt update
apt install nginx php8.2 php8.2-cli php8.2-common php8.2-curl php8.2-fpm php8.2-gd php8.2-igbinary php8.2-intl php8.2-mbstring php8.2-soap php8.2-xml php8.2-zip php8.2-redis
apt upgrade -y
reboot

mkdir build
cd build
sudo sed -i 's/^#\s*\(.*\)/\1/' /etc/apt/sources.list.d/ondrej-ubuntu-php-jammy.list
sudo apt-get update -y
sudo apt-get install -y devscripts build-essential fakeroot
sudo apt-get build-dep -y php8.2-fpm
apt-get source -y php8.2-fpm
cd php8.2-8.2.10
dpkg-buildpackage -rfakeroot -uc

Even running on a hefty VPS with 16x Intel vCPU, 32 GB RAM and a 200 GB NVMe drive, it still took a little over 30 minutes to build, but it was ready. I made the following patch for main/fastcgi.c:

--- fastcgi.c.orig      2023-09-29 10:06:52.180803585 +0000
+++ fastcgi.c   2023-09-29 10:06:15.932395211 +0000
@@ -982,8 +982,10 @@
                if (ret > 0) {
                        n += ret;
                } else if (ret == 0 && errno == 0) {
+                       php_syslog(LOG_NOTICE, "errno 0: %s", strerror(errno));
                        return n;
                } else if (ret <= 0 && errno != 0 && errno != EINTR) {
+                       php_syslog(LOG_NOTICE, "errno non-0: %s", strerror(errno));
                        return ret;
                }
        } while (n != count);

I used strerror() to get the string version of the error number for ease of reading and I also wanted to monitor the progress of reading chunks from the socket too. I made another patch for main/SAPI.c:

--- SAPI.c.orig 2023-09-29 12:32:46.020740222 +0000
+++ SAPI.c      2023-09-29 12:34:35.769896620 +0000
@@ -248,6 +248,9 @@
                SG(post_read) = 1;
        }

+       php_syslog(LOG_NOTICE, "Read %zu bytes (" ZEND_LONG_FMT " bytes remaining)",
+                       read_bytes, SG(request_info).content_length - SG(read_post_bytes));
+
        return read_bytes;
 }

@@ -285,6 +288,7 @@
                        }

                        if (read_bytes < SAPI_POST_BLOCK_SIZE) {
+                               php_syslog(LOG_NOTICE, "POST data reading complete. Size: " ZEND_LONG_FMT " bytes (out of " ZEND_LONG_FMT " bytes)", SG(read_post_bytes), SG(request_info).content_length);
                                break;
                        }
                }

Now it was just a case of running make to incorporate my changes and overwrite the existing binary with my changes.

cd fpm-build
make
systemctl stop php8.2-fpm.service
cp ./sapi/fpm/php-fpm /usr/sbin/php-fpm8.2
systemctl start php8.2-fpm.service

All I had to do then was cause the issue and check syslog for the errors. I've tidied up a snippet here and removed unrelated entries.

Sep 28 21:13:30 test-server php: Read 16384 bytes (1043937 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (1027553 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (1011169 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (994785 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (978401 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (962017 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (945633 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (929249 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (912865 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (896481 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (880097 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (863713 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (847329 bytes remaining)
Sep 28 21:13:30 test-server php: Read 16384 bytes (830945 bytes remaining)
Sep 28 21:13:30 test-server php: errno 0: Success
Sep 28 21:13:30 test-server php: Read 0 bytes (830945 bytes remaining)
Sep 28 21:13:30 test-server php: POST data reading complete. Size: 229376 bytes (out of 1060321 bytes)
Sep 28 21:13:30 test-server php: JSON decode failed: 1060321 229376

Gotcha! 😎

But we didn't get everything?

At least now I can finally understand what's happening. Because PHP thinks it is hitting EOF, before it should, it's beginning to process the request without having the entire payload. What surprises me is that PHP knows it hasn't got all of the request payload, but it continues anyway. This is the part where I can't decide what it should do, but, saying "hey, 50% of this payload is missing, let's go!", is what caught me out here.

To try and determine why the truncation is happening at a particular size, I'd spotted that sapi_read_post_block() was using SAPI_POST_BLOCK_SIZE defined as define SAPI_POST_BLOCK_SIZE 0x4000, or, converting that to decimal, 16,384... Going back to the previous blog post, the POST payload was consistently being truncated at 65,536 bytes, which is 16,384 * 4 = 65,536, so a possible cause for the specific size of the truncation? I'm not so sure, because, when using TCP sockets, tweaking the values for /proc/sys/net/ipv4/tcp_rmem will change the size of the truncation, and the truncated size is not always a multiple of 16,384. Also, when we switched to the Unix socket from the TCP socket, the size of the truncated payload changed to 229,376, which closely aligns with /proc/sys/net/core/[rw]mem_default defined here. I even patched PHP again to modify SAPI_POST_BLOCK_SIZE but that didn't have an impact on the size of truncated payload. At this point, it's still not clear, but I know PHP will read from the socket until all of the data is read, what changes is how much data is there.

Once the read() system call in safe_read() returns 0, we're going all the way back up the call stack to SAPI_POST_READER_FUNC() before any checking of the payload size takes place. In this function, there are only three checks taking place, and only one relates to the size of the payload (source).

if ((SG(post_max_size) > 0) && (SG(read_post_bytes) > SG(post_max_size))) {
    php_error_docref(NULL, E_WARNING, "Actual POST length does not match Content-Length, and exceeds " ZEND_LONG_FMT " bytes", SG(post_max_size));
    break;
}

This seems like a sensible sanity check that the size of the POST payload is not exceeding the post_max_size limitation, which defaults to 8M if not set, but the error message threw me a little. The first part of the error message states Actual POST length does not match Content-Length, which is precisely the error I'm hoping to catch, but I can't see how the conditions match the text of the message, so this seems like a mistake in the text. Based on the conditions, a more accurate error text would be POST length exceeds post_max_size, but I have to let them off as this code is 23 years old! The size of the POST is also checked earlier in the same function (source), before the body is read, by checking the declared content-length against post_max_size, so the check is backed up later after reading the body, and, the error text here is accurate.

if ((SG(post_max_size) > 0) && (SG(request_info).content_length > SG(post_max_size))) {
		php_error_docref(NULL, E_WARNING, "POST Content-Length of " ZEND_LONG_FMT " bytes exceeds the limit of " ZEND_LONG_FMT " bytes",
					SG(request_info).content_length, SG(post_max_size));
		return;
	}

After that, we can run back up through the rest of the call stack and find no additional checks to match the size of the POST that was read against the declared size of the payload. The request will be processed even though we didn't get all of the request.

Is this a bug?

I've certainly gone backwards and forwards on my answer to this question, but ultimately, I think this is a bug. It boils down to choosing between two options:

1. Should PHP dump the partial request with an error?
2. Should PHP attempt to process a partial request?

I have looked at this from a security angle, and, so far, I've not found a way this can be obviously abused, but it depends if receiving partial request payloads would be dangerous for your application. Arguably, this is something you should be handling anyway.

To see behaviour closer to what I was expecting to see, although not quite, I created a patch for SAPI.c that will dump the whole request body if we get a partial request.

--- SAPI.c.orig 2023-09-29 12:32:46.020740222 +0000
+++ SAPI.c      2023-09-29 20:28:36.530234632 +0000
@@ -288,6 +288,12 @@
                                break;
                        }
                }
+
+               if (SG(read_post_bytes) != SG(request_info).content_length) {
+                       php_stream_truncate_set_size(SG(request_info).request_body, 0);
+                       php_error_docref(NULL, E_WARNING, "POST length of " ZEND_LONG_FMT " bytes does not match declared Content-Length " ZEND_LONG_FMT " bytes; all data discarded", SG(read_post_bytes), SG(request_info).content_length);
+               }
+
                php_stream_rewind(SG(request_info).request_body);
        }
 }

It's not perfect, but it does give me some clear feedback as to what happened, and, I don't receive a partial payload to process now. You can see the problem is detected on the final line of this log slice.

Sep 29 20:23:31 test-server php: Accepted connection: 0ce264eae07198c3c59ae90d04127c39
Sep 29 20:23:31 test-server php: Read 16384 bytes (1024737 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (1008353 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (991969 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (975585 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (959201 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (942817 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (926433 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (910049 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (893665 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (877281 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (860897 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (844513 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (828129 bytes remaining)
Sep 29 20:23:31 test-server php: Read 16384 bytes (811745 bytes remaining)
Sep 29 20:23:31 test-server php: errno 0: Success
Sep 29 20:23:31 test-server php: Read 0 bytes (811745 bytes remaining)
Sep 29 20:23:31 test-server php: POST data reading complete. Size: 229376 bytes (out of 1041121 bytes)
Sep 29 20:23:31 test-server php: POST length of 229376 bytes does not match declared Content-Length 1041121 bytes; all data discarded

I guess the only thing left to do is to raise a bug (done) and see what see happens, but for now, at least I finally understand why PHP was trying to process those truncated JSON payloads! 🎉

Outstanding Questions

The final piece of the puzzle that I'm still trying to put together is how we end up in this situation. How does PHP end up reading only part of the request?

My knowledge on the inner workings of TCP sockets or Unix sockets is basically nothing and I've learnt the little that I do know between this blog post and the last one. My current theory looks something like this, and bear in mind, this is mostly guess work!

1. Nginx receives the full POST request from the client.
2. Nginx opens the socket to PHP and starts writing until the buffer is filled?
3. PHP is busy so no child process can spawn to handle the request?
4. Nginx times out and goes away.
5. PHP finally gets around to processing the request, which is limited to the partial content in the buffer.
6. PHP processes the partial request.

Without knowing more on how sockets work, a large portion of my assumptions might be wrong, so it'd be great to hear from someone with real knowledge in this area. If you can help out, please do drop by the comments below and give me some tips or answers!

Another huge thanks has to go to João Poupino again for assistance with this post, it simply wouldn't have been possible otherwise 🍻

This blog post is going to detail what was a pretty lengthy journey for me in debugging an elusive issue that started to occur on Report URI recently! It required me to investigate and learn about things that were outside of my area of expertise, created heaps of frustration and ultimately, of course, turned out to be my fault... 😅

Report URI

I'm sure that many regular readers will be familiar with Report URI, the security monitoring and alerting platform that I founded and operate. Well, recently, I started to observe a sporadic issue that took me a little time to nail down. First, a little on our infrastructure to set the scene.

1. Data is sent by web browsers as a POST request with a JSON payload.
2. Requests pass through our Cloudflare Worker which aggregates the JSON payloads from many requests, returning a 201 to the client.
3. Aggregated JSON payloads are dispatched to our origin 'ingestion' servers on a short time interval.
4. The ingestion servers process the reports into Redis.
5. The 'consumer' servers take batches of reports from Redis, applying advanced filters, threat intelligence, quota restrictions and per-user settings, before placing them into persistent storage in Azure.

The problem was occurring on our ingestion servers when we attempted to parse the JSON payload in the POST requests sent to our origin.

JsonException #3

The problem started with log entries of an exception coming from our ingestion servers. It seems fairly innocuous and, apparently, we were getting some invalid JSON payloads.

Nette\Utils\JsonException #3
Control character error, possibly incorrectly encoded

After inspecting the POST payload in the exception log, it was indeed truncated, causing the Json:decode() to throw an exception.

I was concerned that the exception log may be truncating the POST payload, as indicated at the end of the line, so I enabled some additional logging when handling the JsonException to log the value of the content_length header ($_SERVER['CONTENT_LENGTH']) and the size of the raw POST body that PHP is seeing (strlen(file_get_contents('php://input'))). Here are the results.

[2023-09-02 17-54-26] Problem parsing JSON, content length: 261671
[2023-09-02 17-54-26] Post size was: 65536
[2023-09-02 17-54-27] Problem parsing JSON, content length: 236481
[2023-09-02 17-54-27] Post size was: 65536
[2023-09-02 17-54-27] Problem parsing JSON, content length: 237498
[2023-09-02 17-54-27] Post size was: 65536

It does seem that the declared content length is much larger than what PHP is receiving, and it's not a misleading exception because of truncation in the exception logging, we are actually seeing a truncated payload. It's also interesting to note that the request body is always truncated at 65,536 bytes regardless of how big the actual payload is.

Checking Nginx

My next step was to go upstream and add some additional logging to Nginx which is our web server / proxy in front of PHP-FPM. I wanted to see whether I could observe the same truncated payload behaviour between Nginx and PHP to narrow down where the issue was happening. Here's the Nginx log format I used.

log_format post '$time_local: req_len $request_length - con_len $http_content_length';

This gave me the following log entries when I enabled logging on the ingestion servers.

02/Sep/2023:18:13:24 +0000: req_len 205 - con_len 453
02/Sep/2023:18:13:27 +0000: req_len 35266 - con_len 60944
02/Sep/2023:18:13:27 +0000: req_len 389327 - con_len 923803
02/Sep/2023:18:13:27 +0000: req_len 262352 - con_len 1355353
02/Sep/2023:18:13:27 +0000: req_len 207 - con_len 509168

I'm using the $request_length variable in Nginx which is the "request length (including request line, header, and request body)" and the $http_ variable which allows you to log out an "arbitrary request header field", such as $http_content_length. As you can see, the declared content length is again much larger than the actual request body received. There is always a slight discrepancy here as the $request_length includes the request line and headers, along with the body, and $http_content_length is only the body, but even factoring that in, the difference is significant.

Checking the Cloudflare Worker

At first glance, the logs above lead me to our Cloudflare logs which we collect via Log Push, but I was conscious that in this scenario it felt far more likely that I was at fault than them. Still though, I'd gone from PHP, to Nginx and now the next upstream step was to the Cloudflare Worker.

After trawling through our logs for the corresponding time that we were getting the JSON exceptions (which had stopped of their own accord after ~45 mins), I couldn't find anything that stood out other than Cloudflare receiving 500 errors from our origin, which was to be expected. Not finding anything particularly useful, it looked like Cloudflare was sending us the requests with a proper payload. I added some additional logging to our Worker so that if the issue were to return, we'd potentially have some extra information to work with.

Many of the issues I expected we might see should already be logged because Workers will log all uncaught exceptions, but it's easy to add some console.log() calls which will also be captured (source). I also added some additional request headers on requests to our origin so we could log those and compare metrics like the size of the payload before sending to the declared content-length and then the observed content-length.

  return {
    "X-Request-Body-Length": requestBodyLen.toString(),
    "X-Distinct-Reports":  distinctReports.toString(),
    "X-Total-Reports": totalReports.toString(),
  }

I then decided to do some more investigation across our origin servers to look for clues because I still wasn't sure what had happened.

CPU overload!

I took a look at our DigitalOcean dashboard to see how the ingestion servers were holding up during the window that the JSON exceptions were being thrown, and it seems that they were more than a little busy!

Yikes... This seemed odd because the JSON parse was failing, so I'd have assumed that they were doing significantly less than usual because the problematic inbound requests effectively leave them with nothing to do... During the same time window I could also see elevated Load, elevated RAM usage (but still plenty to spare), no noticeable disk activity (so we weren't paging) and no noticeable change in our ingress (so we didn't have a traffic spike). Our egress on the ingestion servers did display a drop, but that makes sense because there will be less traffic going to Redis if we're not processing as many reports. What on Earth is chewing up that much CPU?

Resource exhaustion

I was caught up on the CPU being slammed at 100% utilisation, especially when we weren't processing our usual volume of reports. After more research online and more debugging, I took a look at syslog and scrolled back to the time window of the issue and found the following entries.

Sep  2 20:22:14 report-ingestion-60 kernel: [12697.456786] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:14 report-ingestion-60 kernel: [12697.485758] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:14 report-ingestion-60 kernel: [12697.520596] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:14 report-ingestion-60 kernel: [12697.826360] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:14 report-ingestion-60 kernel: [12697.973408] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:14 report-ingestion-60 kernel: [12698.075398] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:14 report-ingestion-60 kernel: [12698.183949] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:15 report-ingestion-60 kernel: [12699.001482] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:15 report-ingestion-60 kernel: [12699.073934] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:15 report-ingestion-60 kernel: [12699.148919] TCP: out of memory -- consider tuning tcp_mem
Sep  2 20:22:19 report-ingestion-60 kernel: [12702.594396] net_ratelimit: 42 callbacks suppressed

Huh! This at least gave me something to work with so it was off to do some reading up on exactly what this error messages means, thinking of possible causes and investigating what a solution might look like.

TCP: out of memory -- consider tuning tcp_mem

We're delving outside of my area of expertise now, but I needed to identify the cause of the issue and put a solution in place so that it wouldn't happen again. I started off by looking at what tcp_mem is in the Linux manual and I got the following:

              This is a vector of 3 integers: [low, pressure, high].
              These bounds, measured in units of the system page size,
              are used by TCP to track its memory usage.  The defaults
              are calculated at boot time from the amount of available
              memory.  (TCP can only use low memory for this, which is
              limited to around 900 megabytes on 32-bit systems.  64-bit
              systems do not suffer this limitation.)

              low    TCP doesn't regulate its memory allocation when the
                     number of pages it has allocated globally is below
                     this number.

              pressure
                     When the amount of memory allocated by TCP exceeds
                     this number of pages, TCP moderates its memory
                     consumption.  This memory pressure state is exited
                     once the number of pages allocated falls below the
                     low mark.

              high   The maximum number of pages, globally, that TCP
                     will allocate.  This value overrides any other
                     limits imposed by the kernel.

This seems straightforward enough and the values are not something that I've looked at before. We use the vanilla Ubuntu 22.04 (LTS) x64 image from DigitalOcean when creating all of our servers, so the values would still be whatever the defaults were.

$cat /proc/sys/net/ipv4/tcp_mem
22302   29739   44604

Given that the unit of measurement here is units of system page size, we need that to calculate total size values.

$getconf PAGESIZE
4096

So, 4096 bytes per page gives us the following.

22302 pages x 4096 bytes = 91.348992 MB
29739 pages x 4096 bytes = 121.810944 MB
44604 pages x 4096 bytes = 182.697984 MB

I have no idea if these values are suitable or even if/how they should be tuned, so it was time for another venture down a rabbit hole of research!

After reading countless blog posts on the topic, I wasn't sure if I knew more or less than I did before... There seem to be countless ways to tackle the problem of tuning these values and people have different opinions on how to calculate the new values too. Another problem that I had was, even if I did tune them, I'd need to come up with a way to reproduce the issue to test the fix, and I didn't know what was actually causing the problem yet either.

What is using TCP resources?

Given that the problem revolves around resource exhaustion, specifically TCP memory resources, one solution could be to simply 'add more cloud'. Spreading the load over more servers should certainly help to reduce resource issues, but it's not the right solution. I took a look at what might be using TCP resources on these systems to see if that might point me to something.

1. The Cloudflare Worker initiates TCP connections to our origin to send HTTPS requests.
2. PHP connects to Redis using a TCP socket because the Redis Server sits on our private network.

That didn't seem like a particularly large amount of load but then, Michal Špaček (former Report URI staffer and someone I'd reached out to for help), asked me to check how Nginx was talking to PHP. Turns out we were still using the default configuration that has been there since the dawn of Report URI, and that was a TCP socket! So, now, our list of things putting pressure on TCP resources is.

1. Cloudflare Worker --> TCP --> Ingestion Servers
2. Nginx --> TCP --> PHP
3. PHP --> TCP --> Redis

That seems like a really unnecessary overhead and one that's easy to remove. I switched PHP over to using a Unix Socket instead of a TCP Socket, deployed it on a canary server and waited to see if the change was good. Here's the config change that this needed along with a service reload for both Nginx and PHP-FPM.

nginx v-host config
- fastcgi_pass 127.0.0.1:9000;
+ fastcgi_pass unix:/run/php/php8.2-fpm.sock;

php-fpm config
- listen = 127.0.0.1:9000
+ listen = /run/php/php8.2-fpm.sock

The change seemed good with no adverse effects so I deployed it across the entire fleet, not just the ingestion servers. Everything is managed with Ansible so making changes like this across a large number of servers is really easy.

Then it happened again...

Shit... Having alleviated what should be approximately 1/3 of the load on TCP resources, I was surprised to see this happening again. I wanted to do some investigation while the problem was happening, and fortunately it happened while I was at my desk, so I dug in. I'd also deployed some better monitoring to the fleet in the form NetData Monitoring so that I'd have much more detailed information if the issue came back. (thanks for the suggestion João Poupino, who was now also involved in helping out!)

The issue was much shorter in duration this time, but I had a lot more logging and data ready to help me. It seems that PHP was chewing through all of the CPU resources on the servers.

There was also a huge spike in open sockets (file descriptors) coming from Nginx.

This means that Nginx is backlogging requests and that's likely because PHP is stalled out and I suspect we've hit pm.max_children = 64 which is what we use in production. The 64 PHP processes are busy, no more can spawn, and Nginx is just going to wait to timeout on the fastcgi_pass causing it to backup on incoming connections. Zooming in, we can see that PHP starts to consume sockets first, but then it plateaus, and shortly after that, nginx goes wild and starts massively ramping up on open sockets.

Seems like a solid theory, but, why? What could PHP possibly be doing, especially as the same truncation of POST requests is happening so it should be doing basically nothing!

Looking at the logs more closely

As you saw above, the issue started and stopped within ~1 minute and I didn't change or do anything to impact that. It also happened across our entire fleet of ingestion servers during the same window, give or take a second or two, which makes this more interesting and was suggesting an external factor to the ingestion servers. We centralise all of our logs to Papertrail so I could go and take a time-slice for these two events and look at everything that happened across all of our servers. Of course, there were many of the expected log events with the TCP: out of memory -- consider tuning tcp_mem happening across both incidents and the improved JsonException logging giving me more information too. As before, it seemed like the requests coming from Cloudflare were doing just fine and then, when they hit our origin, the problems begin.

This was when I spotted something that I'd seen before in the logs during the first incident, but noticed that these entries were also present in the second incident too. I'd been focusing on the ingestion servers but I now noticed that some of our other servers were throwing a very small number of seemingly unrelated errors at the same time. I widened the time-slice to see if these other servers were throwing the errors before and/or after the incident and it turns out, they weren't.

Sep 18 15:28:36 report-consumer-65 info.log [2023-09-18 15-28-36] We lost a batch (actually 10): cURL error 7: Failed to connect to reporturi.table.core.windows.net port 443 after 444 ms: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://reporturi.table.core.windows.net/$batch  @  CLI (PID: 1657091): /var/www/html/report-uri/public/www/index.php Process  @@@  git:40620dc

Our consumer servers sit behind Redis and pull out batches of reports to process into Azure Table Storage, and they were receiving a variety of errors when talking to Azure. There were connection timeout errors, connection refused errors and then successful connections, but a 503 Service Unavailable response. It seemed odd that these issues were starting and stopping during the same time window that our ingestion servers were having their issues, even though the two are isolated and independent from each other. I hadn't thought to look much into our Redis server because it already has some very tight monitoring and alerting setup, and I hadn't observed any issues with it throughout. There was a minor drop in traffic from the ingestion servers during the incident, again expected, but other than, it was doing just fine. I then noticed something else in the logs.

Batches of Rejected Reports

When the Cloudflare Worker talks to our origin, there are three endpoints that reports can be sent too. We have batch_reports() which is for a batch of reports to be processed as normal into your account and visible in the Reports/Graphs pages, we have batch_wizard() which is for a batch of reports that were sent to the CSP Wizard, and then batch_reject() which is for reports that the Worker rejected and only need to be counted in your Rejected Reports metrics but not otherwise processed. The vast majority of the exceptions being thrown when parsing the JSON were for requests sent to the batch_reject() endpoint, the one that seemingly has the least to actually do! I decided to walk through the code and refresh my memory because, after all, this was written quite a number of years ago. Here are the three methods, nice and simple:

	public function batch_reports(): void
	{
		$this->checkRequestIsPost();
		$this->batch(RedisKeys::REPORTS());
	}


	public function batch_wizard(): void
	{
		$this->checkRequestIsPost();
		$this->batch(RedisKeys::WIZARD());
	}


	public function batch_reject(): void
	{
		$this->checkRequestIsPost();
		$rawBody = $this->httpRequest->getRawBody();
		try {
			$rejects = $rawBody ? Json::decode($rawBody, Json::FORCE_ARRAY) : null;
		} catch (JsonException) {
			Debugger::log('Problem parsing REJECT JSON, content length: ' . $_SERVER['CONTENT_LENGTH']);
			Debugger::log('Post size was: ' . strlen($rawBody));
			$rejects = null;
		}
		if (is_array($rejects) && count($rejects) > 0) {
			$this->reject->toRedis($rejects);
		}
		$this->output->set_status_header(201);
	}

As you can see, batch_reports() and batch_wizard() go straight to Redis, but batch_reject() has the extra bit of logging I'd added, yes, but it puts these reports into Redis using $this->reject->toRedis() instead. This is when I quickly started to unravel things.

A quick history lesson

The Cloudflare Worker used to buffer and pass reports to our ingestion servers, where the existence of the user would then be checked and the reports processed if a user existed to receive them, or rejected otherwise. This was a lot of load on the ingestion servers at what is the highest velocity segment of our processing pipeline, so, ~5 years ago we built a new feature so the Worker could maintain state about our users. By giving the Worker the ability to query our origin and check for the existence of a user, and then caching the result locally, we could have the Worker discard reports much sooner in the process. This is where batch_reject() came in above, using slightly different code.

We also wanted to avoid hitting Table Storage so aggressively to query the existence of a user so we implemented a service that would maintain a 'User Entity Cache' in Redis and keep it aligned with Table Storage. Now, services could query Redis instead of Table Storage and we saw some enormous performance improvements and a reduction in load on Table Storage as a result. Over time, more and more areas of our code were migrated to use this new User Entity Cache and we now have a `User Entity` service that will transparently query Redis first, and then gracefully fall back to Table Storage if needed, populating Redis in the process.

Except, this old rejected report code wasn't using that process... Calling $this->reject->toRedis() still has a direct dependency on Table Storage, despite what the name implies.

One query to rule them all
One query to find them
One query to bring them all
And in the darkness bind them

Our batch_reject() code was still using a fairly legacy approach to handling these reports and would simply grab the list of our users from Table Storage and run the list of rejected reports against the user list, counting rejected reports where users existed and discarding them otherwise. This worked pretty well, and has actually worked fine up until now. Table Storage is fast, queries are cheap, and sure, it's a bad approach, but it's never been a problem. But, we're a lot bigger now with a lot more users than before. This is where the problem came in.

A query to grab the list of all our users used to be quite a small one. Heck, when we started there was only one user in there! Over time though, that number of users has increased to hundreds, then thousands, and then tens of thousands, and now, we're closing the gap to six figures. It's become an increasingly larger query over time and, because we're talking to Table Storage, I also need to update my TCP overheads list from earlier.

1. Cloudflare Worker --> TCP --> Ingestion Servers
2. Nginx --> TCP --> PHP
3. PHP --> TCP --> Redis
4. PHP --> TCP --> Azure Table Storage

When I thought I'd reduced our overheads on TCP by 1/3, it was only a 1/4 reduction, and it means that those error logs from our consumer servers are now relevant. If they're having issues talking to Table Storage, then the ingestion servers probably are too.

Breaching Azure Table Storage Performance Limits

Azure Table Storage is fast, really fast, but everything has limits, and we were pushing the limits of Table Storage. The limit for Entities Per Second set by Microsoft is defined as:

The scalability limit for accessing tables is up to 20,000 entities (1 KB each) per second for an account.

A couple of things stood out to me right away here. First of all, we have tens of thousands of users, and many servers performing that query, so it's not going to be hard for us to hit 20,000 entities queried per second, but there's also a size limit. If it's 20,000 entities per second, with a 1 KB size, I assume that gives us a theoretical maximum read rate of 20 MB/s, otherwise the size wouldn't need to be specified. Our typical user entity isn't too much larger than this, at 4-5 KB, but we're definitely going to see reduced entity query rates if the size is a factor.

I went over to the Azure Portal and there is a metric available for our storage account to look at the amount of throttled transactions against our tables, and...

We were being throttled by Azure! I'm not sure of the specifics around how this works, and I was seeing connection timeout, connection refused and 503 errors in our logs, but those things would make sense if we were approaching the performance capabilities of Table Storage. The two larger spikes to the right also align with when I observed the issue happening, but not the other spikes. Perhaps they weren't big enough to trigger the backlog problem and we were able to ride it out, but this felt like I was on to something.

Exponential backoff

I dug into our code again to see what the behaviour would be in this scenario, where connections or transactions against Table Storage are failing, and we're using an exponential backoff. That's a sensible approach, and maybe our values need tweaking, but it's not a bad setup.

	private static function pushMiddlewares(ServiceRestProxy $service): void
	{
		$service->pushMiddleware(RetryMiddlewareFactory::create(
			RetryMiddlewareFactory::GENERAL_RETRY_TYPE, <-- retry type
			3, <-- number of retries
			1000, <-- interval in ms
			RetryMiddlewareFactory::EXPONENTIAL_INTERVAL_ACCUMULATION, <-- accumulation method
			true, <-- retry on connection failures
		));
		$service->pushMiddleware(new SanitizeExceptionMiddleware());
	}

With the connection timeouts/failures, or slow responses from Table Storage to get the 503, taking up to 1,000ms or more, coupled with this retry logic, there are easily scenarios here where PHP could be tied up for 15+ seconds. That's a lot of waiting around, and still only a good theory at this point, so it was time to test it.

Reproducing the issue locally

Because all I had was a theory, I wanted to reproduce it and see it with my own eyes. Given the relative simplicity of this problem, it shouldn't prove too hard to reproduce, so I fired up a server with the same resources as our ingestion servers and created a simple PHP script.

<?php

  $fp = fsockopen("scotthelme.co.uk", 80, $errno, $errstr, 30);

  sleep(30);

  $json = json_decode(file_get_contents('php://input'));

  if ($json == null) {
    syslog(LOG_WARNING, "JSON decode failed: " . $_SERVER['CONTENT_LENGTH'] . " " . strlen(file_get_contents('php://input')));
  }

The script needed to simulate what our ingestion servers were doing so it would:

1. Receive an HTTP POST request with a JSON payload.
2. Nginx would pass that to PHP via a Unix Socket.
3. PHP opens a TCP connection to pressure TCP resources.
4. PHP sleeps to simulate slow external connections.
5. Try to parse the JSON payload.

In order to hit this endpoint with load, I created a simple bash script on my local.

#!/bin/bash
curl --location 'http://167.99.196.71/' \
-H 'Content-Type: text/plain' \
-X POST \
-d @post.data \
--insecure

The post.data file contained 1MB of valid JSON so I had a sizeable POST request to test with and then, I just needed to load up the server with requests.

$ for i in $(seq 50); do ./post.sh & done

This would fire up 50 instances of the script at the same time and the command can be run sequentially on a time interval to progressively load up the server. It didn't take long...

Sep 18 22:57:41 delete-me kernel: [ 6910.201032] TCP: out of memory -- consider tuning tcp_mem
Sep 18 22:57:41 delete-me kernel: [ 6910.275034] TCP: out of memory -- consider tuning tcp_mem
Sep 18 22:57:41 delete-me kernel: [ 6910.624642] TCP: out of memory -- consider tuning tcp_mem
Sep 18 22:57:44 delete-me kernel: [ 6913.017896] net_ratelimit: 9 callbacks suppressed
Sep 18 22:57:44 delete-me kernel: [ 6913.017900] TCP: out of memory -- consider tuning tcp_mem
Sep 18 22:58:01 delete-me ool www: JSON decode failed: 1001553 65536

The exact same problem!! The payload was even truncated at the exact same size, which perhaps makes sense because the server has the same resources so probably the same default values around tcp_mem. This was great to see. I could reliably reproduce the problem which means that now it's identified, the fix should be good to go.

Fixing the issue

By now, it's pretty clear what the fix is and it didn't take much effort to implement the changes.

The first and immediate change was for the batch_reject() code to use the User Entity Cache service for querying the existence of users. This should all but eliminate any querying against Table Storage and even if queries were triggered, they would be to query the existence of a single user and not to query out our entire user list.

Quite a small change to make quite a big impact!

My next change will be to shift this code from the ingestion servers to the consumer servers instead. The consumers have far less pressure and given the volume of reports flowing through the ingestion servers, they just need to get out of the way and push reports into Redis as efficiently as possible. That's a ticket for another day, though, because the main focus was to avoid seeing this issue again and I'm happy to say that since the fix was deployed, it hasn't happened again. Not only that, I'm seeing a reduction in the transactions against Table Storage and generally lower CPU usage across the ingestion servers. All is good.

Phew!

That was quite the post, and I hope it was interesting to follow along! Even though the issue was finally resolved, I did still have a few questions left outstanding after this and maybe someone out there can help?

Should Nginx pass these requests through?

I never fully investigated the cause of the truncated payloads because that seemed to be a symptom of the problem rather than the problem itself. My current theory is that the inbound connections were being terminated and the partial payload received was then being processed by Nginx. In this scenario, I was surprised that Nginx would pass it through to PHP, but maybe there's a good reason for that and someone can share it below! I'd have thought that requests like this would not make it beyond Nginx and they'd show up in the error log.

Why is PHP processing these requests?

This is dependent on the answer to the above, but there is also the possibility that Nginx is sending the full request through, but I don't believe so based on the logs from Nginx, and that PHP is cutting short the payload when receiving it. Either way, I'm not sure why PHP would try to process only a partial request whether it was cut short from Nginx or PHP.

How should I tune tcp_mem?

I've read a lot about how to do this but can't say I've walked away with a definitive answer, if one even exists. Our ingestion servers have more than enough resources and no other burdens, so for now, I've given a modest increase in the pressure and high values, but they could probably do with having a little more thought put into them.

I think I've had enough of fighting fires so I'll get back to writing code and building new features 😎

Update 2nd Oct 2023: I've published a second blog post with further debugging and information - Processing Truncated Requests? A PHP Debugging Deep Dive

Over the weekend, I saw a tweet from Troy Hunt who posed a little project idea. Having heaps of spare time... I thought I'd take on the challenge and see if I could help!

The idea

As I'm sure many of you know, Troy runs Have I Been Pwned, a site that tracks data breaches and allows people or organisations to be alerted to their exposure. Of course, after a data breach, one might wonder how seriously a website takes security, given the event.

Looking for a little project to keep you busy on the weekend? I was just thinking: how many of the breached websites in @haveibeenpwned now have a security.txt file? So, if you feel like grabbing those domains and querying them all, there's an API here: https://t.co/ftiKkfH7Hp

— Troy Hunt (@troyhunt) July 23, 2023

That sounds simple enough, let's get on it!

Crawler.Ninja

I have another project called https://crawler.ninja, where I scan the top 1,000,000 sites in the World every day and analyse various aspects of their security. You can see a summary of the daily scan data, or, access a dump of all historic scan data, but be warned, there are now many terabytes(!) of historic data!

I sliced out a bit of the crawler code because I already look for the presence of the security.txt file that Troy was asking for, and then I could run the scan against the list of breached domains from HIBP. Pulling the list of domains for all the breaches in HIBP is easy, here's my code to do it.

<?php

$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, 'https://haveibeenpwned.com/api/v3/breaches');
$response = curl_exec($ch);
$breachList = json_decode($response, true);
$domainList = [];
foreach ($breachList as $breach) {
	if (isset($breach['Domain']) && trim(strtolower($breach['Domain'])) !== '') {
		$domainList[] = trim(strtolower($breach['Domain']));
	}
}
$domainList = array_unique($domainList);
foreach ($domainList as $domain) {
	file_put_contents('output.txt', $domain . "\r\n", FILE_APPEND | LOCK_EX);
}

Here's the list of 641 unique domains that I found and that I used for analysis.

Security.txt

The idea of the security.txt is really quite simply and it's explained in that blog post, but the TLDR; you can put contact info in a text file in this specified location for people to reach you when things go bad. You can see I have one of these files on my important sites:

https://scotthelme.co.uk/.well-known/security.txt
https://securityheaders.com/.well-known/security.txt
https://report-uri.com/.well-known/security.txt

There are some specific requirements on where to host this file and how to present it in RFC 9116, but for the most part, it seems that people get it right. Notably, the following:

For web-based services, organizations MUST place the "security.txt" file under the "/.well-known/" path

The file MUST be accessed via HTTP 1.0 or a higher version, and the file access MUST use the "https" scheme

It MUST have a Content-Type of "text/plain" with the default charset parameter set to "utf-8"

This field [Contact] MUST always be present in a "security.txt" file.

It seems the biggest failure of sites that attempt to meet the requirements is to have a content-type of text\plain instead of the required text\plain; charset=utf-8. For this analysis, I'm going to take a slightly more relaxed approach and allow text\plain but I've also provided the list of compliant sites with the more strict check too.

On the strict check, only 6 sites out of the 641 checked have a security.txt file and on the relaxed check, it's slightly better at 11... That means we're only seeing around 1.7% of these sites using security.txt files!

Security Headers

Whilst I was doing this security analysis, I figured I could quickly and easily extend this to add some more value. Regular readers will know of my Security Headers project that recently joined Probely, where you can do a free security scan of your website in ~2 seconds! Well, Security Headers also has an API so I wrote a quick API client to run the list of HIBP domains against the Security Headers API too. You can see the raw results to do your own analysis, but I think this graph nicely summarises that things aren't as good as they could be.

There were a handful of domains that don't resolve, some that blocked our scanner outright and some that didn't respond to the scanner, but we got a successful scan on 483 domains show in the data. Here's how the scores break down for them.

Surprisingly it's not too bad, but there was only a single site that got an A+, on a list of a some really, really big sites.

It will be interesting to see how this tracks in the future and whether or not the presence of something like a security.txt file, or even your score on Security Headers, can be used in any way as indicator of how seriously you take security! 🤔

Want to try the Security Headers API? Get 10% off your first 3 months with HIBP10 at checkout!

As I sit and write this blog post I still find it absolutely unreal how far this little idea, that I had all of those years ago, has come! Let's take a look back at the journey of Security Headers so far, and the journey ahead.

Humble Beginnings

All the way back in Feb 2015, I was really starting to dig into the analysis of Security Headers on other sites and after getting bored of digging around in Dev Tools, I created a tool to make it easy for me. Outlined in my blog, Introducing SecurityHeaders.io, I launched the first version of the site, which looked very different to how it does today!

There was no scoring, no nice layout, it was simple and basic but it got the job done.

Less than a year later, though, in Dec 2015, I published Launching the new version of securityheaders.io, which introduced scoring for your A+ to F grade and the visuals that will be familiar with you today. The scoring was inspired by SSL Labs after I'd noticed how much people will 'chase the grade'. If you tell someone they got a grade B, they almost naturally want to improve that, and I wanted to harness that same gamification that SSL Labs had for SSL configuration and put it to good use for Security Headers configurations!

Just a few months later and Security Headers made it to the front page of Hacker News and things really began to take off! It could have been the gamification of the grading, or just a friendly user sharing a link for us, but it resonated well with the community and we got a huge swell of support.

A day on the front page of Hacker News and @securityheaders has just passed 250,000 scans!!! pic.twitter.com/ED7qlqDKix

— Scott Helme (@Scott_Helme) February 11, 2016

250,000 scans was a really big deal for me back then, and it's pretty wild to think that we've added another 3 zeros since, but I had absolutely no idea that this was only the beginning!

Continuing To Grow

As the months and years ticked by, we continued to cross through some awesome milestones. As each one came and rolled by, I still couldn't believe just how popular the site was becoming and the site seemed to be growing in popularity at  a relentless pace.

Overnight we passed through 1,000,000 scans! 🎉🎉🎉 pic.twitter.com/lh6PR1Ds56

— Security Headers (@securityheaders) July 8, 2016

We passed through 10,000,000 scans!!! 🎉🎉🎉 pic.twitter.com/NGnYZqJyBP

— Security Headers (@securityheaders) February 15, 2018

We're so grateful to have @probely sponsoring us and supporting the project! As we fast approach 100,000,000 scans, having their support is essential to keep the service free and available for everyone. Check them out here: https://t.co/x8xVG5TjW7 ❤ pic.twitter.com/6C4LXCWLLZ

— Security Headers (@securityheaders) September 2, 2020

Very quickly we hit 100,000,000 scans in Sep 2020 and I really felt like I'd made something to be proud of. One of the most notable memories I have around that time was of an old colleague and friend sharing a penetration test report with me that they'd received and in it, a screenshot from the Security Headers site!! Their guidance was that they needed to improve their HTTP Response Headers and Security Headers had established itself as such a reputable player in the industry, they were happy to refer to us as their proof with a grade F! If you've got any similar stories, or places that you've seen Security Headers linked or referenced, please let me know in the comments below, it'd be awesome to see.

In that very same month, we also announced our newest sponsor, Probely, who were one of only a few companies to ever come forwards and support this free tool used by so many. This sponsorship would turn out to be our longest standing, and most supportive, eventually culminating in an even larger announcement. The growth of the site continued and Security Headers added more powerful capabilities and became yet more popular. As the awareness around Cyber Security continued to rise, or as more people just shared a link to this free tool, the numbers grew and grew.

You can see the clear upward kick in our scan numbers from Feb 2019 onwards, and whilst I don't know exactly what happened to cause that, it's been a mega journey to see it not only grow the way it has, but to continue to maintain that growth too.

To The Future

Regular readers will know that just a few weeks back, I announced that Security Headers is joining Probely. That blog post outlines all of the details so you can head over there if you want the lowdown, but one of things that I said in that announcement was that Security Headers would live on exactly as you knew it before. And it has.

Security Headers is continuing to see the same rate of growth under Probely that it did previously and we're currently working on things behind the scenes to make that even better. My previous blog post on Security Headers announced our new API so you can easily, and cheaply, automate the regular scanning of your websites and I'm also really happy to see great growth there too.

All in all, I couldn't be more pleased with how things are working out for this little tool that I built to make my life a little easier and then decided to share with the World! Hopefully I'll see you back here in 2024 for the 300,000,000 scans announcement 😎

We've encountered a lot of problems of our own making in the TLS/PKI ecosystem in recent years, and whilst we've got better at dealing with them and even avoiding them, there's still a way to go.

Certificate Lifetime

The focus of these blog posts will be on the maximum allowed validity period of certificates, but not just the certificates used by websites, we'll be taking a look at CA certificates too. To get started, I'll be looking at the certificates that almost all of us use, and that's the certificates we all get so we can have HTTPS on our websites! I'll refer to them throughout as server certificates, but you may have come across them being called HTTPS certificates, SSL certificates, end-entity certificates, or one of a few other terms too!

Let's have a brief look at the history of certificates and if we roll back the clock far enough, there was a time when there wasn't a defined cap on the maximum validity period of a certificate. Before July 2012 you could go wild and it's easy to come across certificates that were issued with a validity period of 9 or even 10 years! [1][2]

Thinking about that now is absolutely mind blowing and we've chopped that down to just a fraction of what it used to be, and here's how we got here.

60 Months

The first time a limit was set on the validity period of a publicly-trusted certificate was in v1.0 of the Baseline Requirements. In §9.4 of that very first document, you can find the following:

Certificates issued after the Effective Date MUST have a Validity Period no greater than 60 months.

With an effective date of 1st July 2012, we saw the introduction of a limit that reduced certificates to being valid for "only" 5 years... That's pretty wild, and still completely inappropriate by today's standards, but that was a reduction from the 9 or 10 years you could see back then.

39 Months

The next reduction in certificate lifetime came on 1st April 2015 in v1.3.0 of the Baseline Requirements, and that saw the follow stipulation in §6.3.2:

Certificates issued after 1 April 2015 MUST have a Validity Period no greater than 39 months

A step in the right direction for sure, but this was still far too long, and there are many good reasons why, a few of which I detail here.

825 Days

As time progressed, we saw the next reduction in certificate lifetime come on 1st March 2018 in v1.4.4 of the Baseline Requirements, which set out in §6.3.2:

Certificates issued after 1 March 2018 MUST have a Validity Period no greater than 825 days

At this stage, certificates are getting to close to what has always been my recommendation of 12 months being the absolute limit, but we started to encounter some real friction from here.

398 Days

The industry tried to push through the idea of 398 day certificates before we had 825 day certificates, but the vote failed. Ballot 185 called for 398 day certificates from 24th August 2017 and was met with widespread resistance from major CA players in the industry. The 825 day certificates vote was seen as the compromise and passed, but it was inevitable that 398 day certificates would be proposed again.

I gave widespread coverage to the next ballot, SC22, which again proposed that certificates be reduced to 398 days in validity and again, failed. The reasons presented by those in the industry who wanted this change were all valid and they sought to resolve genuine concerns, I even covered many of them myself in 'Why we need to do more to reduce certificate lifetimes'. Having failed twice to shorten certificates, a key player in the ecosystem was to step up and in the interests of improving security for their users and single-handedly push through this change.

It was a surprise to see Apple announce this, but their 'About upcoming limits on trusted certificates' is short and sweet.

TLS server certificates issued on or after 1 September 2020 00:00 GMT/UTC must not have a validity period greater than 398 days.

Published on 11th March 2020, the announcement gave the industry 6 months of advanced warning and showed that Apple was serious about improving the health of the ecosystem. Shorter certificates were coming, and while Apple was making it happen, Google and Mozilla stood with them. With this change now being effectively a mandatory change, ballot SC31 'Browser Alignment' solidified the 398 day validity period in the Baseline Requirements V1.7.7 §6.3.2:

Subscriber Certificates issued on or after 1 September 2020 SHOULD NOT have a Validity Period greater than 397 days and MUST NOT have a Validity Period greater than 398 days.

Where to next?

For a long while now, I've been eagerly awaiting the announcement of the next reduction in certificate lifetime and it hasn't yet arrived. I think the pandemic certainly didn't help, but still, I feel we're now overdue for the next step.

Keeping a close eye on things happening in the industry, I have seen a few changes that could be interpreted as an indication that another announcement is definitely coming, and they also hint at what that next step might be.

Certificate Transparency Policy

If you aren't familiar with Certificate Transparency, I have an introductory blog post that should help you get started. To give a TLDR; here, CT logs are public logs that contain all certificates so that the existence of any certificate can't be hidden and certificates can't be issued in secret. The number of logs a certificate must be written to depends on how long the certificate is valid for, with longer certificates needing to be written to more logs.

Both Apple and Google have had their own CT policy requirements for some time now, and they looked like this.

As you can see, the longer the certificate is valid for, the more logs it has to be written to. Of course, as time went by, the longer certificates simply didn't exist and the policy became mostly redundant, to be replaced with this.

Logging to CT logs creates a burden for the issuing CA and it's a burden that they're likely to want to minimise. By introducing these new requirements, both Apple and Google are giving CAs a benefit to achieve by issuing shorter certificates and this is one of the things that first got me wondering if this was their way of hinting at what was coming next. Drawing the line at 180 days would certainly seem to fit well with our progress on shortening certificates, and this could be a good way to get CAs and subscribers (the people acquiring certificates) used to a 180-day cadence. Then, Google gave another hint.

Chrome Root Program

Much like Mozilla run their own Root Authority Program for their products like Firefox, so too do Google with Chrome and other major players in the industry with their own Root Authority Programs for their own clients like Microsoft and Apple. The Chrome Root Program, sadly not named the Chrome Root Authority Program (CRAP) as would it would otherwise be, published Moving Forward, Together, a post worth reading for many reasons, but I'll focus on just one of them here.

a reduction of TLS server authentication subscriber certificate maximum validity from 398 days to 90 days

90 days?! Before we get too excited, it's worth noting that the section I quoted that text from begins with:

In a future policy update or CA/Browser Forum Ballot Proposal, we intend to introduce

This is Chrome laying the groundwork for the next change in certificate validity periods, but it doesn't exclude another step between where we are now and when 90 days becomes the norm.

What's the next reduction?

Personally, I'm torn between what's the right option for the next change, and that might come as a surprise to some given my views and the blog posts I've published over the years!

On one hand, the answer is both easy and obvious, it should be 90-day certificates! On the other hand, trying to be a little more pragmatic, one has to wonder if the industry is quite ready for 90-day certificates...

My biggest hesitation is the low number of CAs that support ACME, the protocol that allows easy and standardised automatic renewal of certificates. I've detailed the few that do offer free certificates via ACME, but at this point, I don't understand why all CAs don't support ACME, including the commercial ones.

If we're going to 90-day certificates, the process of renewal simply must be automated. There's no way that it would be reasonable for anyone to consider renewing those certificates manually because they should be renewed every 30 days, or maybe 60 days at the absolute most. Once I think that, though, I then wonder how 180 days would make any difference... Would people renew those certificates manually? Is that still a reasonable expectation or should 180-day certificates also be renewed in an automated fashion? If that's the deciding factor then we should just go to 90-day certificates as everyone will need to automate anyway.

Maybe another way to look at this is to look at the reductions in certificate validity over time, and then plot both 90-day or 180-day certificates as the next change. If we do that, you can see that one of these follows the trend much more nicely and seems like the more logical choice. The following graphs show the certificate validity limit in months on the Y axis and the date the change came into effect on the X axis. The graphs assume the next change will be introduced in Jan 2024.

Here is the graph with 180-day certificates.

Here is the graph with 90-day certificates.

You can see that the trend line is following much better when the next change is to 90 days and the 180 day change is really flattening the bottom of that slope. On top of that, you can see that 90 day or 180 day are both, quite clearly, a reduction in our rate of progress over time and that's assuming the change comes in Jan 2024! If we push this out further than that, which is almost a certainty at this point, things only start to look worse.

Other Considerations

The most obvious consideration for this change is the impact on subscribers, those using the certificates for HTTPS. It means more frequent renewals of certificates, more frequent deployments of certificates, and possibly implementing a whole new set of technologies and processes if you're doing manual renewal at present. There are some other considerations too though and I thought I'd list them here briefly.

Can CAs handle the load?

Every certificate that is issued requires a CA to go through the issuance process and that requires the appropriate amount of infrastructure. If we reduce certificate validity periods, then CAs will have to complete that process more frequently, even without increasing their number of customers. For a CA issuing only 10,000,000 certificates per year, they'd be doing ~27,400 issuances per day at present assuming 1-year certificates. If we go to 90-day certificates, that same CA would now need to handle ~333,300 issuances per day, quite an increase!

Considerations on this load increase would need to made for their HSM capabilities to do the signing operation, database activity, storage for logs, bandwidth both internally and externally, along with much more. There aren't many orgs out there that have the ability to do >10x on their production load on short notice! You can read this article from Let's Encrypt on their concerns with having to reissue all of the non-expired certificates that they have, something which is a slightly different concern, but mirrors all of the same performance and infrastructure worries. If you look at the Let's Encrypt stats, however, they're comfortably issuing >3,000,000 certificates per day without any problems so it can be done, the CAs might just need to make some improvements.

Can CT Logs handle the load?

I briefly mentioned Certificate Transparency at the start of this blog post and if you're unfamiliar with how it works, it requires that CAs log all certificates they issue to a minimum of two independently operated CT Logs. A lower lifespan on certificates of course means that more entries will need to be made into the CT Logs because more issuance events are taking place, and thus an increase in the associated costs of operating the log. This will require more bandwidth, more computational power and more storage, at a minimum, from all log operators! We've already dealt with some quite significant increases in the load place on CT logs and Temporal Sharding, as explained here by Venafi, should be good enough to keep the sheer storage requirements at bay, but it doesn't solve the other concerns like bandwidth and compute power. Here are some details on the Let's Encrypt CT Log which are almost 4 years old and based on 1,000,000 certificates being issued per day, so imagine where we are now as Let's Encrypt are comfortably doing 3,000,000 certificates per day!

We use 2x db.r5.4xlarge instances for RDS for each CT log. Each of these instances contains 8 CPU cores and 128GB of RAM.

We use 4x c5.2xlarge EC2 instances for the worker node pool for each CT log. Each of these instances contains 8 CPU cores and 16GB of RAM.

A back of the napkin storage estimation is 1TB per 100 million entries. We expect to need to store 1 billion certificates and precertificates per annual temporal shard, for which we would need 10TB ... We decided to create a 12TB storage block per log (10TB plus some breathing room)

With a global issuance rate of ~250,000 new certificates per hour, a rate that is only growing, CT Log Operators will certainly have some interesting times ahead!

90-day certificates

I've long pointed out the need for shorter certificates and once the process of issuance and deployment is automated, the validity period of a certificate no longer matters. All of the certificates that I use, both internally and externally, are automatically renewed and deployed, so I could renew them every 7 days if I really wanted to and apart from changing the frequency of the renewal task, I wouldn't have to lift a single finger. The big push here isn't really about the validity period of a certificate, it's a push towards automation, and once we get that widespread, we'll be in a much better place!

If you enjoyed this blog post and would like to dive deeper into the World of TLS and PKI, why not join me on our Practical TLS and PKI workshop!

I'm super excited to be making this announcement for a whole bunch of reasons that I'll go into in detail below, but, the headline is that Security Headers will be joining Probely!

The Announcement

For anyone who uses the site or the API, this won't change anything, but for me, this will bring positive changes that I wanted to share with everyone.

Probely have been sponsoring Security Headers since Sep 2020(!) and are our longest standing sponsor. The relationship has worked so well because we're aligned in a few different ways. Getting a DAST scan from Probely is a logical next step after a free Security Headers scan, we're both focused on providing value within the community, they're a great bunch of technical people and I get along well with everyone I've met there.

If you look back at the history of sponsors on Security Headers, I've always taken care to ensure that our sponsors are aligned with what we do, and it's never been the case that you can simply pay to be a sponsor. There had to be more to it than that, and that's why there were periods with no sponsor at all and I funded the project myself. To now take things to the next level, we'll be bringing Security Headers and Probely a little closer together!

Why the change?

In may ways, I can summarise this decision with "it felt right". Security Headers wasn't for sale and I wasn't looking to sell it, but as we started to explore how we could work more closely together and the great things we could do, it naturally became the obvious path forward for us to join forces.

Probely will now take care of Security Headers and I will continue to provide the same input and direction for the site. Now though, instead of it being me doing this alone, I'll have a great team of people around me to work with, which will be a refreshing experience! Nothing will be taken away from Security Headers and scans will still be free, but if anything happens to me, it can and will continue!

Alongside continuing to maintain Security Headers, I'll also be joining Probely as a Strategic Advisor to contribute in a variety of ways to their product offering too. This is another element of working alongside great people that will be awesome and whilst only a small time commitment, having a team of people sharing the same passion around me is an exciting prospect.

All-in-all, this change wouldn't be visible from the outside but I wanted to be transparent and share this great news with everyone out there. From here, things will only continue to get better!

There's a new feature in Chrome Dev Tools that's going to make it easier than ever to get started with Security Headers like Content Security Policy! Let's take a look at how to override HTTP Response Headers and build a basic CSP in just a few minutes.

Override HTTP Response Headers

The feature that I'm going to be showing here is available in Chrome 113, which at the time of writing I was testing in Beta, but is now available in stable so check if you have an update available. Your Chrome version needs to be >= 113 for the feature I'm showing.

One of the great things about this new functionality in Dev Tools is that not only can you override the value of HTTP Response Headers that exist, you can also set entirely new HTTP Response Headers that don't exist to test out what effect they will have. This is going to make testing certain Security Headers even easier, like Permissions Policy, Cross Origin Embedder Policy and Cross Origin Opener Policy and of course, Content Security Policy. All of these headers do have safe modes for testing, where you can deploy these headers and they won't take any blocking action, and instead provide feedback on what blocking action they would cause, if any. That approach is great, and as I say it makes testing them totally safe, but you will still have to deploy them somehow. This may only be a single line of server config, application code or a tweak in your CDN settings somewhere, but it's a step you need to take nonetheless. Now, you can do really extensive testing of these Security Headers (and more!) from the comfort of your own browser.

Creating an override

For the purposes of this demo, I'm going to be using my own personal site to deploy a Content Security Policy header so I can do some easy testing. Open the site you want to work with, open Dev Tools and head to the Network tab. Once there, find the network request for the page load, click it to open the details, click Headers, and scroll to the Response Headers section. Here, you will see the new 'Header overrides' option.

That will prompt you to create an overrides file, so look for the banner near the top of the browser to give Chrome a location and permission to store the file. Once done, it will take you to the Sources tab where you can view your overrides file and you can edit it to specify your override rules.

I'm going to 'Add override rule' and then add my basic CSP header which would typically make a real mess of the page!

We can now refresh the page and the override rule be active, resulting in, as you'd expect, almost everything on the page being blocked with the associated errors showing up in the Console.

Because the override rule is only being enforced locally on my client, this is a really safe way of seeing exactly what would happen given a particular configuration of your CSP or other Security Headers. What's even better is that the browser will behave exactly as it would behave if that header came from the server, which means all of the features of CSP are available to you, including sending reports!

Creating a CSP the easy way

It was 5 years ago now when we introduced the CSP Wizard on Report URI. The CSP Wizard takes your CSP reports and allows you to build a policy by looking at the items that are present on your site. Looking at the documentation, you can see that a simple CSP-Report-Only header is required.

Content-Security-Policy-Report-Only: default-src 'none'; form-action 'none'; frame-ancestors 'none'; report-uri https://{subdomain}.report-uri.com/r/d/csp/wizard

This policy won't block anything and break the page, because it's in Report-Only mode, but it will cause a report to be sent for every item on the page. That information, the reporting of every asset on the page, is what allows you to quickly and easily audit all of resources loading across your site. Typically, this header would need to be deployed into production, but now, we can deploy it locally and get a huge amount of feedback in just a few seconds.

In the My Policies section of Report URI, I'm going to create a new policy called "Scott Demo" and enable the Wizard.

From the Setup page I can get my unique reporting address that will need to be used in my CSPRO header, and then I'm good to update the header override with the new values.

As you can see, the page is back to normal now because the CSPRO header doesn't cause anything to be blocked, but the console is still indicating all of those CSP errors.

The crucial part here is the CSP reports have been sent as the report-uri directive was set, and you can see the reports in the Network tab.

All I have to do now is head over to the CSP Wizard page and I can see all of the new items that have been detected and need my attention. From here, it's a simple case of deciding if each item is supposed to be there, in which case I can select the item and add it to my CSP, or, if it's not supposed to be there, I select it and block it, meaning it won't be added to my CSP.

You'll need to click around a few pages on your site to make sure you're exercising all of the functionality, and by just skipping around a few pages, I managed to get a few more items reported. For example, my Disqus comment system only loads on blog posts and doesn't load on the homepage, for obvious reasons. I also only have YouTube videos embedded in certain blog posts so only those pages would report them. This is why we generally advise to deploy the CSP Wizard in production for a day or so, but in just a few seconds, I've gathered a heap of feedback about the resources on my site.

After approving the appropriate entries in the CSP Wizard, I now have the following in My Policies:

This is already a good chunk of the way towards generating a viable policy to use on my site. What I'd do now is replace the policy in the header override rule with the one above, so those items are no longer being reported, reducing the volume of reports being sent, and continue browsing around. Eventually, you would be best deploying this onto the live site, but the majority of the work has now been done in just a few minutes locally.

I think these header overrides are going to be really useful for people to play around with and test features like CSP without having to get their hands too dirty to start with. If you want to give the CSP Wizard a go, or try out any of our many features, you can sign up for a 30-day free trial over at Report URI to get started!

It's been a really long time coming, but, the end is finally here for the padlock icon in the address bar! 🔒🚫

A Long Road

Wow, where do I start?! Whilst the dawn of the encrypted Web was in 1994, with the release of SSLv2.0, the real transition to an encrypted Web didn't start until ~2014, a little after the Snowden revelations. If we look at the % of page loads that have used HTTPS on the Web, you can see when we started putting effort into the problem.

Note that we don't really have any reliable data prior to 2013 because nobody was paying attention, so that section of the graph is my best estimation, but from 2013 onwards we have multiple, reliable sources of data and the graph is accurate.

Along the way to an encrypted Web we've come across HTTPS Anti-Vaxxers, shorter certificates, warnings on HTTP sites, the removal of the EV UI, and countless other major changes in the industry that I've documented well here on my blog. But now, the pièce de résistance, the padlock indicator, is to be retired after almost 30 years of service. To be clear, this was always inevitable. We've seen the removal of the word 'Secure', removal of the green colour, removal of 'https://' in the address bar and now, the removal of the padlock icon. The writing has been on the wall for a long time and there are now a lot of people out there who owe me $1 for losing our bet that 2023 would be the year the padlock was removed!

An Update on the Lock Icon

The blog post from Chromium, An Update on the Lock Icon, is short and sweet. It details everything that you'd expect from them, including research and links to sources for data, something that other industry players never seem to be able or willing to provide. But, without further ado, let me introduce you to the replacement for the padlock icon!

This will be hitting desktop variants of Chrome from ~September 2023 and will be following on Android, but not iOS. For iOS, the padlock icon will be removed and will not be replaced by anything. The reasons for choosing this particular icon given by Chromium are:

We think the tune icon:

• Does not imply "trustworthy"
• Is more obviously clickable
• Is commonly associated with settings or other controls

Whilst I can agree with those reasons, I think one of the most important sentences in the blog post is this one:

Replacing the lock icon with a neutral indicator prevents the misunderstanding that the lock icon is associated with the trustworthiness of a page, and emphasises that security should be the default state in Chrome.

The issues of "trust", and what that word really means, along with other confusions around connection security indicators have long plagued the Web, but no more. In addition, the pursuit of a default secure world is one that simply cannot be argued with, and anything that moves us towards that being a reality, including our UI state, is one I'm onboard with. I've enabled the new indicator in my browser just to see what it looks like and, I have to say, it convinced me even more that this is a step in the correct direction.