A story about how incident response training went wrong, with valuable lessons about pod priorities, isolation, and the importance of a healthy incident response culture.

Our alerting system needs to tell us when it's broken. How incident.io pages its own engineers if the product itself goes down.

Of the mental models and rules I use in my life, by far the most useful is to learn only one thing at any given time.

My reflections on 2023, now my second full year at incident.io. Doubled the team this year (34 to 77), launched Status Pages and Catalog, and spent the last six months building a really exciting new product.

Our CTO offered to upgrade our MacBooks if we could prove it was worthwhile, so we rebuilt a Go hot-reloader, instrumented developer builds and analysed compiler performance until we had an answer.

We build lots of UIs inside Slack. Slack previews let us collaborate on designing those experiences much more quickly.

Whenever a system has access to a consistent store, you can extend that consistency through compare-and-swap to the system's users. This post shows how you can add CAS to an HTTP API using example code and real-world examples.

If you build a state machine on top of a relational database you can abstract concurrency problems away from your business logic and allow developers to write safe-by-default code without dealing with concurrency concerns.

This post explains how to build a library that offers those protections, and how they work under-the-hood.

How incident.io configures its app from a mix of environment variables, config files and secrets stored in Google Secret Manager.

From the moment you learn programming people tell you "don't repeat yourself!"

So what I'm about to suggest might sound odd. But I'm here to say that if you want to ship high-quality software at pace, you should be investing in abstractions that are designed to enable copy-and-paste.