You may have seen the news recently about Twitter and GitHub having accidentally logged sensitive information like passwords in their applications or server logs. Logging is great and it provides valuable information but it has to be done with caution. Other site operators should take note of this event and make sure that it can't happen to them, which is exactly what we did with Report URI.
The news
There were a couple of news stories recently about both GitHub and Twitter logging their user's passwords by accident. From a technical standpoint I can understand how this happens and with such big name companies in the headlines, it could clearly happen to any of us. It was only a few weeks ago when the story about GitHub broke and the story about Twitter broke too. I do wonder if one of these prompted the other in much the same way it prompted us to check out Report URI and make sure it couldn't happen to us. We log a lot of things from HTTP server logs and application error/exception logging to the logs with our SaaS/PaaS providers too. We log a lot of things and we don't want passwords, or any other sensitive data, to be one of those things.
Report URI
As soon as I saw the stories about GitHub and Twitter I immediately thought about Report URI. If it could happen to companies like them it could happen to a company like us and I couldn't, without checking, be 100% sure we would never do this, so I raised a bug to find out.
The first course of action was to search all of our logs to see if anything sensitive had been logged. We use PaperTrail for logging so this was pretty easy and to be honest I needed to either put my mind at rest or confirm we had an issue as soon as possible. I am really happy to report though that nothing sensitive had been logged and we didn't need to switch to panic stations. The next thing on the list was to figure out if something could be logged. Just because we had nothing sensitive in the logs it didn't mean that it couldn't be logged, just that we might have been lucky and not had anything logged. Turns out, there was a little luck involved...
Michal has been working on Report URI for a while now and in his investigation he did find we could log sensitive information when an exception happened. It was only times we have a plaintext password, like account creation / password reset / password change, and the most prominent scenario would be if Azure was down so we couldn't contact Table Storage which we use to store all of our data. In these scenarios if we threw an exception for any reason the error log would contain the POST parameters of the request which of course would contain the user's password in the clear. This was not acceptable and we worked on getting a fix developed, tested and deployed as quickly as possible.
Depending on your application you may have other things you consider to be sensitive and not want logging. For us the first and most obvious one was passwords, and probably will be for most sites, but we also have the user's 2FA secret which we are also applying protections to. We don't have to worry about credit card data as we fully outsource all processing to Stripe and never actually have it to log but that could be another consideration for you. Now, if an exception does happen in any circumstance where we have the POST data containing a password, the field will be replaced by special handling.
We can now rest assured that no matter what happens we've taken steps to prevent us ending up with sensitive data in our log files.
Learn from other people's mistakes
GitHub and Twitter are huge organisations that have a security budget which is probably many times larger than the total value of our company. That doesn't mean that bad things can't happen to them and if something does happen to them, we should really take a hint and check our own apps too. No organisation is perfect, and that includes us, so all we can do is learn and improve. Nothing bad came from this for us as no sensitive data was ever logged, but we're now more secure than we were before and are being open about our journey in the hope of prompting others to check for similar issues within their own applications and infrastructure. I think the wider lesson to learn here is pay attention to what's happening to others and learn from their mistakes where possible, it's much more preferred compared to learning from your own.