Alerts That Don't Wake You at 3 AM

Most on-call misery is caused by bad alerting, not bad systems. The pager that goes off at 3 a.m. for a CPU blip nobody needed to act on, the alert that fires every deploy, the twenty warnings everyone mutes, that is the actual problem, and it is fixable. Good alerting follows a short set of rules: alert on symptoms users actually feel, page a human only when a human must act now, and ruthlessly delete everything else. Do that and the pager becomes trustworthy again, which is the entire goal.

The counterintuitive truth is that fewer alerts make a system safer, because every noisy page erodes trust until the real one gets ignored too.

Why alerting quality is a reliability issue

The pager is the interface between your systems and a human’s sleep. If it fires accurately, on-call is sustainable and real problems get fast attention. If it fires constantly for things that do not matter, two things happen: people burn out, and, worse, they start ignoring the pager, so the one alert that actually mattered gets muted along with the noise. Bad alerting does not just annoy; it makes the system less reliable.

This is why alerting deserves the same rigor as the code it watches. It is the last link in the observability chain, the point where all your metrics, traces, and logs turn into a human taking action, and a broken last link wastes everything upstream. This post is part of the Observability series and pairs directly with SLOs and Error Budgets That Hold Up.

What makes a good alert?

A good alert is actionable and urgent and tied to real user impact. Actionable means when it fires, there is a specific thing a human should do; urgent means it needs doing now, not tomorrow; user-impact means it reflects something a customer actually experiences, not just an internal number crossing a threshold. An alert that fails any of these three tests should not page anyone.

The three-question filter for any page:

• Is it actionable? If there is nothing a human can do about it, it is not an alert, it is a notification. Send it somewhere else.
• Is it urgent? If it can wait until morning, it is a ticket, not a 3 a.m. page.
• Does it reflect user harm? If users are fine, waking someone is by definition a false alarm.

Only alerts that pass all three deserve to page a human. Everything else belongs on a dashboard (for context) or in a ticket queue (for non-urgent follow-up), not on the pager.

Should you alert on symptoms or causes?

Alert on symptoms, the things users actually experience, rather than on every internal cause. Symptom-based alerting (error rate up, latency up, requests failing) catches any problem that affects users, including failure modes you never anticipated, and it produces far fewer alerts than trying to enumerate and alert on every possible internal cause. Cause-based alerting produces a flood of pages, many of which do not actually affect anyone.

This is the same principle behind SLO burn-rate alerting: tie the page to user-facing impact, and you get few, meaningful alerts that correlate with real harm. The internal causes still matter, but they are what you investigate after a symptom alert fires, not what you page on individually.

What is alert fatigue, and why is it dangerous?

Alert fatigue is what sets in when people receive so many alerts, especially noisy, non-actionable ones, that they stop taking them seriously. The danger is not just morale; it is that the genuinely important alert arrives in the same stream as the noise and gets ignored or muted along with it. A noisy alerting system is actively less safe than a quiet one, because it trains its own users to disregard it.

Every false or non-urgent page spends a little of the team’s trust in the pager, and that trust does not come back easily. Once on-call has learned that most pages are noise, they respond slower to all of them, including the real one, which is the exact opposite of what alerting is for. This is why deleting noisy alerts is not just tidying up; it is directly protecting the response time to real incidents.

How do you reduce the number of alerts?

Cut aggressively: delete alerts that have never led to an action, convert non-urgent ones into tickets or dashboard panels, replace cause-based alerts with symptom-based ones, and tie paging to SLO burn rate so only real user harm wakes someone. The target is a small number of high-signal pages, not comprehensive coverage of every metric, because coverage of everything is exactly what produces the noise.

A practical alert-reduction pass:

1. Audit what actually paged last quarter. For each recurring alert, ask: did a human do something because of it? If not, delete or downgrade it.
2. Downgrade the non-urgent. Anything that can wait becomes a ticket or a dashboard, off the pager.
3. Replace causes with symptoms. Collapse many internal-cause alerts into a few user-facing symptom alerts.
4. Route by severity. Page for urgent user harm; use non-paging channels for everything informational.
5. Tie pages to SLO burn rate. Fast budget burn pages now; slow burn is a ticket. This single change removes a lot of noise.

The mindset shift is from “alert on everything so we don’t miss anything” to “page only on what needs a human now, and trust the dashboards and logs for the rest.” Comprehensive alerting feels safe and produces the fatigue that makes you miss the real thing.

What should you do with non-urgent alerts?

Route them off the pager and into a channel matched to their real urgency: a ticket queue for things that need action but not now, and a dashboard for things that are only context. The mistake is treating “worth knowing” as “worth paging,” which is exactly how the pager fills with noise. Most things that fire are worth knowing and not worth waking someone.

A simple routing model by urgency:

• Page now (phone, wake someone): actionable, urgent, user-facing harm. The SLO is burning fast.
• Ticket / async notification (chat channel, work queue): needs a human eventually, but it can wait for business hours. Slow SLO burn, non-urgent degradation.
• Dashboard only (no notification): useful context you look at when investigating, not something that should interrupt anyone.

The discipline is to consciously place every alert in one of those tiers and to keep the top tier small. A capacity warning that is trending but has days of runway is a ticket, not a page. A single retried request is dashboard context, not even a ticket. When you route by real urgency, the pager shrinks to only what genuinely needs a human now, which is what makes it trustworthy. This tiering is the practical mechanism behind the whole “delete noisy alerts” principle: most noisy alerts are not deleted, they are demoted to the tier they always belonged in.

How do you fix an alerting system that is already noisy?

Most teams do not start from zero; they start from four hundred alert rules accumulated over years, most of which nobody trusts. The advice above describes the destination. Getting there from a noisy system needs a different approach, because a rewrite is not on the table.

The method that works is incremental and evidence-driven:

1. Stop the bleeding. Freeze new paging alerts for a period. Every new alert must justify why it pages rather than tickets, reviewed by someone other than the author. Noise accumulates because adding an alert feels responsible and free.
2. Rank by volume. Pull the last 90 days of pages and sort by rule. The distribution is almost always brutally skewed — a handful of rules produce the large majority of pages. That handful is your entire near-term backlog.
3. Take the top one each week and change it. Raise the threshold, lengthen the duration, downgrade it to a ticket, or delete it. One rule, one week, permanently. This is slow enough to be sustainable and fast enough that on-call improves within a quarter.
4. Delete anything that has never been actioned. A rule that has fired fifty times and never once led to a change is not protecting you. Deleting it is not a risk; keeping it is, because it consumes attention that the real alerts need.
5. Backfill coverage from incidents, not from imagination. For each incident found by a user rather than by an alert, add exactly one alert. This grows coverage where you demonstrably lack it, instead of adding rules for failures that have never occurred.

The reason to attack the noisiest rules first is not that they are worst in principle — it is that they dominate the experience. Fixing the top rule can measurably improve on-call in a single week, which is what earns the effort continued support. Starting with a comprehensive audit produces a large document and no improvement.

One cultural point that determines whether this sticks: deleting an alert must be a normal, low-ceremony act. Where removing an alert requires justifying that the failure it watched can never happen, nothing is ever deleted and the system only accretes. The correct default is that an alert which has not proven its value gets removed, and can be added back if a real incident argues for it.

How should alerts be routed and escalated?

Deciding what deserves an alert is half the work. The other half is where it goes, and a good alert delivered to the wrong place at the wrong urgency is functionally a bad alert.

Start by separating severity from destination, because conflating them is what produces systems where everything pages:

Tier	Condition	Destination	Response time
Page	Users are harmed now and a human must act	Phone, waking someone	Minutes
Ticket	Real problem, no user harm yet, will need work	Issue tracker	Days
Notify	Informational; useful context during an incident	Chat channel	None guaranteed
Dashboard only	Diagnostic signal	Nowhere	Pulled, not pushed

Most alerting systems are unhealthy because everything landed in the first row by default. The migration that fixes it is mechanical: take each existing alert, ask the three-question filter, and move it down the table. Teams routinely find that a majority of paging alerts belong in rows two or three, and that moving them changes nothing except that people sleep.

A few routing rules that prevent the common failures:

• Route to the owning team, not to a shared firehose. An alert everyone receives is an alert nobody owns — the same ownership problem that afflicts services, and it has the same fix: a declared owner.
• Escalate on silence, not on severity. If the primary does not acknowledge within a set window, page the secondary automatically. Relying on a human noticing that nobody responded defeats the purpose.
• Group related alerts into one notification. When a dependency fails, twenty services may alert simultaneously. Twenty pages for one incident is the fastest possible route to alert fatigue; deduplicate by incident, not by alert rule.
• Suppress downstream alerts when an upstream cause is already firing. If the database is down, alerts for every service that depends on it add nothing and bury the one that matters.
• Always have a maintenance window mechanism. Without it, planned work generates pages, and people learn to ignore pages during deploys — which is exactly when a real problem is most likely.

The escalation detail most often missed: the escalation path must be tested. A secondary who has never been paged, a rotation with a stale phone number, or an integration that silently stopped delivering are all discovered during the incident when they are discovered at all. A quarterly test page costs nothing and reliably finds at least one broken link.

How do you measure whether your alerting is good?

Alerting quality is usually discussed as taste. It is measurable, and measuring it is what turns “we should clean up our alerts” into work that actually happens.

Four numbers, reviewed monthly:

Metric	How to compute it	A healthy value
Actionability rate	Pages that led to a human action ÷ total pages	High. Anything below about half means most pages are noise
Auto-resolve rate	Pages that resolved with no intervention ÷ total pages	Low. These are pure noise and should become tickets or be deleted
Pages per on-call shift	Count per rotation	Low enough that a person sleeps. Two or more per night is unsustainable
Detection coverage	Incidents found by an alert ÷ all incidents	High. The inverse — found by a user or by luck — is your gap

The last two are in tension, and that tension is the whole discipline. Cutting pages is trivial if you accept missing incidents; catching everything is trivial if you accept waking someone hourly. Tracking both stops you from optimising one into the ground.

Auto-resolve rate is the most useful and least tracked. An alert that fires and clears itself before anyone opens a laptop has, by construction, never needed a human. Either its threshold is too tight, its duration too short, or the condition is genuinely self-healing. Every one of those has a fix that is not “page someone.”

Two habits that make the numbers move. Review the noisiest alert every week and change something about it — threshold, duration, severity, or delete it. And require a runbook link on every paging alert: alerts that nobody can write a response for are usually alerts that need no response.

What makes a runbook useful at 3 a.m.?

An alert’s value is capped by what the responder can do with it, and most runbooks are written for a reader who is calm, informed, and at a desk. The actual reader was asleep four minutes ago.

Write for that person. A runbook that works under those conditions is short and ordered by what to do, not by how the system works:

• What is broken, in one sentence, from the user’s point of view. “Checkout is failing for some customers,” not “error rate on svc-payments exceeded 2%.”
• How to confirm it is real, in one or two steps. The first thing a responder needs is to know whether this is a false alarm.
• The mitigation, first and explicit. Roll back, fail over, disable the feature flag, scale up. Mitigation comes before diagnosis — restore service, then investigate. A runbook that opens with a debugging tutorial has the priorities backwards.
• When to escalate, and to whom by name. Include the condition, not just the contact: “if not mitigated in 15 minutes, page the database on-call.”
• What not to do. The destructive action that looks tempting at 3 a.m. and makes things worse deserves an explicit warning.

Two properties separate runbooks that hold up from ones that do not. Every command is copy-pasteable, with no placeholders a tired person must reason about. And it has been used at least once in a drill, because an untested runbook is a hypothesis. Running one against a real alert during business hours reliably exposes the step that assumed context the responder does not have.

The honest note: if writing a useful runbook for an alert turns out to be impossible because there is no clear response, that is a finding about the alert. It should not page. Convert it to a ticket, or delete it.

An alerting checklist

For a pager people can trust:

• Every paging alert is actionable, urgent, and tied to user-facing impact.
• Non-actionable and non-urgent alerts are tickets or dashboards, not pages.
• Alerts are symptom-based (user experience), not one-per-internal-cause.
• Paging is tied to SLO burn rate where possible: fast burn pages, slow burn ticketed.
• You audit alerts regularly and delete any that never lead to action.
• Severity routing sends urgent things to the pager and everything else elsewhere.
• The team trusts the pager, because when it fires, it matters.

What I’d do differently

The mistake I have watched teams make (and lived through) is adding an alert after every incident as a reflex, “we didn’t catch this, so add an alert”, until the pager is a wall of noise nobody trusts. Each addition felt responsible; the sum was an alerting system that made on-call worse and response slower, because the real alert was buried in a hundred that never mattered.

If I were building alerting from scratch, I would start from the user-facing symptoms and the SLO, page only on those, and add cause-level detail to dashboards and logs rather than to the pager. And I would treat alert deletion as routine hygiene, auditing quarterly and cutting anything that never drove an action, the same way you would delete dead code. A pager that fires rarely and always means something is worth more than one that catches everything and is trusted by no one. Fewer, better alerts is not a compromise on reliability; it is what reliability on-call actually looks like.

Sources

• Google SRE Book, Monitoring Distributed Systems (symptom-based alerting): sre.google/sre-book/monitoring-distributed-systems
• Google SRE Workbook, Alerting on SLOs (burn rate): sre.google/workbook/alerting-on-slos
• Google SRE Book, Being On-Call: sre.google/sre-book/being-on-call