A threshold alert fires when a metric crosses a fixed value, such as an error rate above 1%. An SLO-based alert fires when a service is using its error budget too fast for its own SLO. SLO alerts reflect user impact, so they fire less often on noise and catch slow burns that thresholds miss.
What is SLO-based alerting?
SLO-based alerting fires an alert when a service is consuming its error budget faster than its SLO allows. Instead of asking “is the error rate above 1%?”, it asks “at this rate, will the service miss its reliability target?”
The measure it uses is burn rate: the observed error ratio divided by the error ratio the SLO allows. The standard form, multi-window burn-rate alerting, checks a long and a short window together.
Why are fixed error-rate thresholds not enough?
Traditional monitoring frequently uses fixed thresholds. For example, an API might generate an alert whenever its error rate exceeds 1%. Although straightforward, this approach has four limitations.
Problem 1: Not every error rate has the same impact
Consider two services. Service A has a 99% success-rate SLO. Service B has a 99.99% success-rate SLO. Both experience a 0.5% request failure rate.
For Service A, the allowed error rate is 1%, so its current failure rate remains below the SLO’s permitted average. For Service B, the allowed error rate is only 0.01%, so the same failure rate represents a much more serious reliability problem.
The fixed rule treats both the same. Burn rate shows only one is in trouble.
A fixed 1% error-rate alert would treat both services identically. SLO burn-rate alerting normalizes failure rates against each service’s reliability objective, making the resulting signal more meaningful.
Problem 2: Short monitoring windows can generate noise
Suppose an API experiences a brief spike in failures lasting 30 seconds. A monitoring rule using a short observation window might trigger an alert even if the service recovers immediately.
If this happens repeatedly, engineers can experience alert fatigue, making it harder to distinguish meaningful reliability risks from temporary fluctuations.
Problem 3: Long monitoring windows can respond too slowly
Now consider an API experiencing a severe failure rate immediately after a deployment. A monitoring rule using only a six-hour average may take time to reflect the degradation, because earlier healthy periods dilute the observed error rate. During that delay, the service continues consuming its error budget.
Problem 4: Alerts may remain active after recovery
A service may recover from a failure, but its long-window average can remain elevated because the historical failures are still included. This can leave alerts active long after the underlying problem has stopped.
SLO alerts vs threshold alerts: comparison
| Threshold alert | SLO-based burn-rate alert | |
|---|---|---|
| Fires when | A metric crosses a fixed value | The error budget is being used too fast for the SLO |
| Same rule for every service? | Usually | No: scaled to each service’s target |
| Brief spikes | Often fire | Ignored unless the long window is also burning |
| Slow, persistent failures | Often missed | Caught by a low threshold over a long window |
| After recovery | May stay active | Clears once the short window recovers |
| What it tells you | Something changed | The SLO is at risk, and how urgently |
Symptom alerts vs cause alerts
Threshold alerts are often set on causes: CPU, memory, queue depth, a single dependency’s errors. SLO alerts are set on symptoms: what users experience.
Cause metrics are still valuable for diagnosis, on dashboards and in investigation. But as on-call alerts, they fire for problems users never notice and stay quiet for failures nobody predicted. Alerting on the symptom, and using causes to explain it, keeps alerts few and meaningful.
How do you move from threshold alerts to SLO alerts?
- Pick one important service and define its SLI and SLO.
- Add multi-window burn-rate rules for fast, sustained and slow burns. Prometheus burn-rate alerts has ready-made examples.
- Run both side by side for a few weeks and compare which alerts were useful.
- Demote cause-based alerts that never needed action to dashboards or tickets.
- Repeat for the next service.
Frequently asked questions about SLO-based alerting
What is SLO-based alerting?
SLO-based alerting fires when a service is consuming its error budget faster than its SLO allows, measured as a burn rate, rather than when a metric crosses a fixed value.
Why are error-rate threshold alerts noisy?
A fixed threshold fires on brief spikes that recover on their own, and it uses the same number for services with very different reliability targets. Both create alerts that need no action.
Do SLO alerts replace all other alerts?
Not entirely. They should be the main on-call alerts for user-facing services. Some cause-based alerts, such as a disk about to fill, are still worth keeping because they predict failures before users are affected.
What is symptom-based alerting?
Alerting on what users experience, such as failed or slow requests, rather than on internal causes such as CPU or memory. SLO-based alerting is a form of symptom-based alerting.
Key takeaways
- Threshold alerts use one fixed number; SLO alerts are scaled to each service’s target.
- Fixed thresholds are noisy on spikes, slow on long averages and blind to slow burns.
- Multi-window burn-rate alerts fire on real risk and clear after recovery.
- Alert on symptoms users feel; use cause metrics to explain them.