SLOs and Error Budgets That Hold Up

An SLO that nobody enforces is just a number on a wiki, and most SLOs are exactly that. The version that holds up does three things: it is set from the user’s actual experience, it is expressed as an error budget you can spend, and it drives a pre-agreed policy about what happens when the budget runs out. Without that last part, an SLO is decoration; with it, reliability becomes a quantity engineering and product manage together instead of arguing about after every incident.

The reason SLOs matter is that they replace the unwinnable “how reliable should we be?” debate with a number and a rule everyone agreed to in advance.

Why SLOs beat vague reliability goals

“Make it reliable” is not actionable, and “aim for 100% uptime” is both impossible and ruinously expensive. Every nine of reliability costs more than the last, and beyond a point the user cannot even tell. SLOs replace that fog with a specific, agreed target and, crucially, an explicit budget for the failure you are willing to accept.

That budget is what makes reliability a shared, tradeable quantity rather than a source of conflict. It gives engineering permission to ship (you have budget) and a clear signal to stop and stabilize (you are out of budget), which is the whole point of the framework. This post is part of the Observability series and is the reliability-target counterpart to the incident work in Staff engineer craft.

What is an SLO and an error budget?

An SLO (service level objective) is a target for a reliability metric over a window, for example “99.9% of requests succeed within 300ms over 30 days.” The error budget is the inverse, the amount of failure the SLO permits: 100% minus the SLO. A 99.9% SLO grants a 0.1% error budget, which over a month is a concrete, countable allowance of failed requests or downtime you may spend however you like.

The mental shift is from reliability-as-aspiration to reliability-as-budget. You are not trying to never fail; you are given a specific quantity of failure to spend, and the job is to spend it wisely, on planned risk, on deploys, on experiments, rather than blow it all on one unplanned outage. That framing, from Google’s SRE practice, is what makes the rest of the system work.

How do you set a good SLO?

Set the SLO from the user’s experience, not from whatever the system currently does. Pick the handful of signals that actually reflect whether users are getting a working service, typically request success rate and tail latency, and choose a target high enough that users are happy but not higher than you can sustainably meet. Setting the SLO to 100%, or to whatever number the system happened to hit last quarter, are the two classic mistakes.

The discipline is to anchor on the user. What does a working experience mean to them, and what level of failure would they never notice versus find unacceptable? Set the SLO at the line between those, measure it with the right metrics, and resist the urge to add nines that cost fortunes and change nothing for the user.

What is an error budget policy, and why it’s the whole point?

An error budget policy is a rule, agreed by engineering and product in advance, for what happens as the budget is spent. The standard shape: while the error budget is healthy, teams ship features and take normal risks freely; when the budget is exhausted, risky changes halt and the team prioritizes reliability work until the budget recovers. This converts the SLO from a passive dashboard number into an actual decision rule.

This policy is what separates SLOs that hold up from SLOs that are theater. Without it, the budget gets blown and nothing changes, so the number means nothing. With it, the budget creates an automatic, pre-negotiated truce in the eternal features-versus-reliability fight: nobody has to argue during the incident, because the rule was set when everyone was calm. That pre-agreement is the real product of the whole exercise.

What is burn rate, and how do you alert on it?

Burn rate is how fast you are consuming the error budget relative to your window. A burn rate of 1 means you are on track to spend exactly the whole budget by the end of the period; a burn rate of 10 means you are burning ten times too fast and will exhaust the budget in a tenth of the time. Alerting on burn rate, rather than on raw error spikes, is how you page for real problems without paging on every minor blip.

The elegance of burn-rate alerting is that it scales the urgency to the severity:

• Fast burn (a high rate over a short window) means you are about to blow the budget quickly, so it pages someone now.
• Slow burn (a modest rate over a long window) means a steady, low-grade problem eating the budget, so it is a ticket, not a 3 a.m. page.

This is exactly the discipline behind alerts that don’t wake you at 3 a.m.: tie the alert to budget impact, so the page correlates with real user harm rather than with any transient error. Multi-window, multi-burn-rate alerting is the SRE standard for getting this balance right.

What is the difference between an SLO, an SLA, and an SLI?

An SLI is the measurement, an SLO is the target for that measurement, and an SLA is a contractual promise with consequences. The SLI (service level indicator) is the actual number, like the percentage of successful requests. The SLO is the internal goal for it, like 99.9%. The SLA is an external agreement with a customer that carries penalties if breached, and it is deliberately set looser than your SLO.

The relationship is a ladder. You measure an SLI (what is really happening), you hold yourself to an SLO (what you aim for internally), and you may promise a customer an SLA (what you will owe them if you miss). The critical design rule is that the SLA must be more lenient than the SLO, so your internal alarm bells ring well before you breach a contract. If your SLA and SLO were equal, you would only find out you were in contractual trouble at the moment you were already in it.

In practice most teams need SLIs and SLOs and may never have a formal SLA at all, because SLAs are for external commitments with money attached. The internal machinery, measure the SLI, target the SLO, spend the error budget, act on the burn rate, is what actually drives reliability day to day. The SLA, when it exists, sits on top as the customer-facing promise, set with comfortable margin below your internal objective.

How do you pick the right SLI to measure?

The indicator is chosen before the target and matters more, because a well-chosen target on the wrong signal measures nothing useful.

The strongest SLIs share a shape: a ratio of good events to total events, where “good” is defined from the user’s side. That framing is worth insisting on, because it produces indicators that are both computable and meaningful, and it rules out most of the vague candidates.

For a request-driven service, the standard set:

SLI	Definition	Watch out for
Availability	Successful responses ÷ total requests	Decide explicitly whether 4xx counts as failure — usually not, since it is the client’s error
Latency	Requests faster than a threshold ÷ total	Pick a threshold users notice, then measure the proportion under it rather than a percentile value
Correctness	Responses with the right answer ÷ total	Hardest to measure and often the one that matters most
Freshness	Data younger than a bound ÷ total reads	The right SLI for pipelines, caches, and derived views

The latency row contains the subtlety most teams miss. “p99 latency under 300 ms” and “99% of requests under 300 ms” sound identical and behave differently in an error budget: the first is a statistic you compare, the second is a good-events ratio you can accumulate, spend, and alert on with burn rate. Express latency SLOs as proportions, not as percentile thresholds, and the whole budget machinery works on them.

Two rules for choosing among candidates. Measure as close to the user as possible — ideally at the load balancer or from real client telemetry, since anything measured inside the service cannot see requests that never arrived. And prefer few indicators. Two or three per critical journey is plenty; a service with fifteen SLIs has no SLO, because nobody can act on fifteen simultaneous signals and the budget policy becomes unenforceable.

For non-request systems, translate rather than skip. A batch pipeline’s availability is “runs completed successfully ÷ runs scheduled” and its freshness is “outputs produced within the deadline ÷ total.” A queue consumer’s is “messages processed within the lag bound ÷ total.” The good-events-over-total shape generalises further than most teams assume, and reaching for it is usually faster than inventing something bespoke.

How do you choose the right SLO target?

The number itself gets debated far more than it deserves, usually in the abstract. There is a straightforward way to derive it.

Start from what users already tolerate, not from what sounds impressive. Look at your current measured availability over the last quarter. If you have been running at 99.7% and nobody has complained or churned, then 99.9% is an ambitious but defensible target and 99.99% is a fantasy that will consume a year of engineering for benefit nobody perceives.

The cost curve is the argument. Each additional nine is roughly ten times the effort and expense of the one before it:

Target	Downtime per month	What it typically demands
99%	~7.2 hours	Basic monitoring; a single well-run region
99.9%	~43 minutes	Redundancy, fast rollback, real on-call
99.95%	~22 minutes	Multi-zone, automated failover, careful deploys
99.99%	~4.3 minutes	Multi-region, no manual steps in recovery
99.999%	~26 seconds	Extremely few systems genuinely need this

Two sanity checks that resolve most disagreements. Compare against your dependencies: you cannot be more available than the things you depend on multiplied together. Promising 99.99% on top of a cloud database that offers 99.9% is arithmetic that does not work, and writing it down usually ends the debate. And compare against the client path: if users reach you over mobile networks with worse reliability than your target, additional nines are invisible to them and expensive to you.

There is also a case for a target that is deliberately not the highest achievable. An SLO that is always met tells you nothing and constrains nobody. A target set so that you occasionally consume most of the budget is one that is actually informing decisions — which is the entire purpose.

What does an error budget actually buy you in practice?

The error budget is usually explained as arithmetic — 99.9% availability leaves about 43 minutes per month — and the arithmetic is the least interesting part. What the budget actually does is settle an argument that otherwise recurs forever.

Every engineering organisation has a standing tension between shipping features and improving reliability. Without a budget, that argument is resolved by whoever is most persuasive in the room, and it is re-litigated every quarter. With a budget, it is resolved by a number that was agreed in advance, when nobody was under pressure.

That is the real product: a pre-negotiated decision rule. Budget remaining means ship; budget exhausted means stop and fix. Neither side has to win the argument again, because the argument was had once, calmly, and written down.

This also reframes what unspent budget means. A team consistently finishing the month with 90% of its error budget intact is not excellent — it is over-invested in reliability relative to what it promised, and it could be shipping faster or running cheaper. That is genuinely counterintuitive and it is the correct reading: an error budget is a target to spend, not a score to maximise. Consistent underspend means the SLO is set below what users actually need, and the honest response is to tighten it or redeploy the effort.

Three properties separate budgets that work from budgets that are decoration:

• The consequence is agreed in advance and in writing. “We stop feature work” must be committed to before it is needed. Deciding at the moment of exhaustion guarantees an exception.
• It is measured on the same signal as the alerts. If the SLO uses one query and paging uses another, every incident begins by reconciling the two.
• Leadership has agreed. An error budget policy that engineering believes in and management overrides on the first collision is worse than none, because it teaches everyone the rule is theatre.

What are the most common SLO mistakes?

The failure modes are consistent, and most of them come from treating the SLO as a documentation exercise rather than a decision-making tool.

Setting it to 99.99% because it sounds serious. Each additional nine is roughly an order of magnitude more engineering effort and cost, and the honest question is whether users can even perceive the difference. If your users reach you over mobile networks that are themselves less reliable than that, four nines of backend availability is invisible to them and expensive to you.

Measuring from the server rather than the user. A request that never reached your service because a load balancer dropped it counts as success by server-side metrics and as failure by the user. SLOs measured only from inside the system reliably report better numbers than users experience, which is precisely the gap that erodes trust in the SLO.

Averaging across everything. One SLO covering all endpoints lets a healthy high-volume endpoint mask a broken low-volume one. Checkout failing entirely may be invisible in an aggregate dominated by a health-check route. Set SLOs per critical user journey, not per service.

Never revisiting it. An SLO set at launch and untouched for two years describes a system that no longer exists. Review quarterly against actual performance and actual complaints.

Having no SLI you can compute reliably. The indicator has to be measurable from data you actually collect, consistently, without manual steps. An SLO whose measurement is a person running a query is an SLO that will stop being measured within a month.

An SLO checklist

Before an SLO is worth having:

• It measures the user’s experience (success rate, tail latency), not internal system trivia.
• The target is high enough for users, low enough to sustain, and never 100%.
• It is expressed as an error budget you can count and spend.
• There is a written error budget policy both engineering and product agreed to.
• The policy actually changes behavior when the budget is exhausted (features halt, reliability work starts).
• Alerting is burn-rate based: fast burn pages, slow burn ticketed.
• The SLO is reviewed periodically, because the right target changes as the product and users change.

One more consideration that decides whether the SLO survives its first year: the SLI must be computable from data you already collect, automatically, with no manual step. An indicator that depends on someone exporting a spreadsheet each month will stop being measured within two months, and an unmeasured SLO is indistinguishable from no SLO at all. Prefer a slightly less perfect indicator that a dashboard computes continuously over an ideal one that requires a human. Automatability is not a nice-to-have here; it is what separates an SLO that shapes decisions from one that becomes a stale wiki page within a quarter. When in doubt, ship the automated version of the imperfect indicator first and refine the definition later against real data.

What I’d do differently

The failure I have seen most is the SLO that exists purely to have one: a number picked from thin air, put on a dashboard, and never connected to any decision. The budget gets blown every month, nothing happens, and everyone quietly learns the SLO is meaningless. That is worse than no SLO, because it looks like rigor while providing none.

If I were introducing SLOs, I would write the error budget policy first, before the number, because the policy is the part that creates behavior. A modest SLO with a real policy that teams honor beats a perfect SLO nobody acts on. And I would alert on burn rate from day one, so the reliability signal is tied to user harm rather than to noise. An SLO is only as good as the decision it drives, so design the decision first and let the number serve it.

Sources

• Google SRE Book, Service Level Objectives: sre.google/sre-book/service-level-objectives
• Google SRE Workbook, Implementing SLOs and error budget policies: sre.google/workbook/implementing-slos
• Google SRE Workbook, Alerting on SLOs (burn rate): sre.google/workbook/alerting-on-slos