SLOs and error budgets: where SRE starts

Reliability as a measurable target

Site reliability engineering (SRE) is an approach to operations that treats reliability as an engineering problem with measurable targets. It originated at Google, but the core ideas do not require a large organization or a dedicated SRE team. They require agreeing on what "reliable enough" means and measuring it.

Three terms do most of the work. A service level indicator (SLI) is a measurement of behavior users care about. A service level objective (SLO) is the target for that measurement over a period. A service level agreement (SLA) is a contractual commitment, usually with consequences, and is typically looser than the internal SLO.

Pick a few user journeys, not every metric

Start with the two or three things users would be most upset to lose, such as logging in, completing a purchase, or loading a dashboard. For each, choose an SLI that reflects the experience: the proportion of requests that succeed, the proportion that finish faster than a latency threshold, or the proportion of data that is fresh and correct.

sum(rate(http_requests_total{job="api", code!~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))

Set an SLO you can defend

An SLO is a target such as "99.9% of requests succeed over a rolling 30 days". Pick it based on what users need and what the system can sustainably deliver, not on the highest number that sounds impressive. Each extra nine is dramatically more expensive.

• 99% over 30 days allows about 7.2 hours of unavailability.
• 99.9% over 30 days allows about 43 minutes.
• 99.95% over 30 days allows about 22 minutes.

Use the error budget to make decisions

The error budget is the amount of unreliability the SLO permits: 100% minus the target. It turns an argument between shipping features and protecting stability into a shared, visible number. While there is budget remaining, teams can ship and take reasonable risks. When the budget is exhausted, reliability work takes priority until it recovers.

Alert on burn rate, not on every blip

Alerting on raw thresholds such as CPU above 80% produces noise and misses real user impact. Burn-rate alerting asks how fast the error budget is being consumed. A fast burn pages someone because the budget will be gone within hours; a slow burn creates a ticket because it needs attention but not at 3 a.m. Combining a long and a short time window reduces false alarms and still detects real problems quickly.

Learn from incidents without blame

Incidents are inevitable, so the goal is to shorten them and avoid repeating them. Keep a simple incident process with clear roles, communicate status while the incident is open, and write a blameless postmortem afterwards that focuses on contributing factors and concrete follow-up actions rather than individuals.

Reduce toil, then repeat

Toil is manual, repetitive operational work that scales with the size of the service and produces no lasting value. Track it, automate the worst of it, and use the time you recover to improve reliability. Start with one service, one or two SLOs, and a monthly review of how you performed. Expand only when the first loop is working.