The Pager Goes Off and Nobody Wakes Up

It’s 3:12 in the morning and your phone is buzzing on the nightstand.

You know the rest. Ack the page so it stops escalating. Find the laptop. Fight the VPN. Open four dashboards. Join the war room channel, where two other people are also just waking up. Stare at a graph. Guess. Check the deploy log. Guess better. Roll back. Watch the graph. Type “monitoring” in the channel. Go back to bed and don’t sleep.

Here’s that night as a timeline. I’ve seen some version of it hundreds of times.

03:12:00  monitor fires: checkout p99 latency over threshold
03:14:07  coffee maker on
03:19:30  on-call acks the page
03:31:00  on VPN, dashboards open
03:44:00  war room has three people in it
04:02:00  someone spots the 11pm deploy
04:09:00  rollback started
04:16:00  latency back to normal
04:20:00  "monitoring"

Sixty-four minutes from alert to recovery. And it isn’t done. The rollback bought time. Someone still has to find the real bug, write the patch, open the incident and write the postmortem nobody wants to write.

The monitor was never the slow part

I was the second SE at Datadog. I spent years helping teams get better signal: tighter monitors, smarter thresholds, fewer junk pages. That work mattered. But look at the timeline again. The monitor fired at 03:12:00. Everything after that was people.

Detection got fast a long time ago. The gap between “we know” and “it’s fixed” is where the hours go, and that gap is human. Waking up. Getting context. Deciding. I wrote about the four DORA metrics a few years back, and in my experience time to restore was always the one where better tooling helped the least. More dashboards don’t make a tired person think faster.

The new loop

Now run the same night with an agent as the first responder.

The monitor fires and sends a webhook. An agent picks it up. It has read access to the logs, the metrics, the deploy history and the repo. It sees that the latency jump lines up with the 11pm deploy. It rolls that deploy back through the same pipeline a human would use, and it watches the monitor recover.

Then it keeps going. It reads the diff and finds the change that caused the problem. It writes a fix, runs the tests and opens a PR. It opens the incident, writes the timeline from what it actually did, and drafts the postmortem.

In a test rig I built at home, monitor to rollback to fix PR runs in about three to four minutes. The human wakes up at 7, reads the postmortem with coffee and reviews a PR.

That’s not science fiction. Every piece of it is a webhook, an API and a pipeline you already have. What’s new is something that can read all of it at 3am without being tired.

Rollback first, diagnosis second

The order matters. Agents, like people, want to find the root cause before they act. Don’t let them. Undo first.

The biggest reminder of this in recent memory wasn’t an agent at all. It was a bad software update in July 2024. Microsoft estimated that CrowdStrike’s update affected 8.5 million Windows devices, “less than one percent of all Windows machines,” and the takeaway Microsoft gave the industry was to “prioritize operating with safe deployment and disaster recovery using the mechanisms that exist” (Microsoft ).

Less than one percent, and it was still front-page news around the world. How fast you can undo a change matters more than how fast you can explain it.

So the first permission I’d give an on-call agent is rollback through your existing pipeline. Not a hotfix straight to prod. Not hand-editing config on a box. Rollback, the boring way, with the same checks a human rollback gets.

What still belongs to a human

An agent on call doesn’t mean nobody is on call. It means the human’s job changes.

• Merging the permanent fix. The agent opens the PR. A person approves it.
• Talking to customers. Status page wording and account calls stay with people.
• Calling it when the agent is wrong. If the rollback doesn’t fix the graph, the agent should page a human, not try five more things.
• Owning the postmortem. The agent drafts it. The team decides what to change.

The test I use is simple. Would I let a new hire do this alone on their first on-call shift at 3am? If not, the agent doesn’t do it alone either.

How to try it without betting the company

Start small and boring.

1. Pick one service with good monitors and a clean deploy history.
2. Wire one monitor to the agent. Just one.
3. Give it read access everywhere and write access to exactly one thing: rollback.
4. Run it in shadow mode. It writes down what it would have done, next to what the human did.
5. Compare the two timelines for a month. If the agent’s is right and faster, give it the real rollback.
6. Add the fix PR and the postmortem draft after that. Both are low risk because a human reviews both.

Shadow mode is the step people skip. It’s also the step that gets your SREs on board, because they get to grade the agent before they have to trust it. Nobody likes a new teammate who shows up and starts pushing buttons on day one.

Watch the monitors too. An agent that answers every page will answer the bad ones as well. If half your pages are noise, you’ll find out fast. That’s the same lesson I wrote about in revisiting observability : signal quality is the whole game.

MTTR is becoming a systems metric

For most of my career, MTTR measured people. How fast they woke up, how well they knew the system, how good the runbook was. Teams improved it with training, rotations and better dashboards.

That’s changing. When the first responder is software, MTTR measures how ready your system is for an agent. How good your monitors are. How clean your deploy history is. How much you can trust your tests, because the fix PR is only as good as the tests that passed on it.

That last one is a bigger deal than it sounds, and it’s the next post.

Your pager is still going to go off. The question is who answers it.