Prometheus burn-rate alerts: PromQL and recording rule examples

In Prometheus, a burn-rate SLO alert divides the rate of bad requests by the rate of all requests over two windows and fires when both are above the SLO’s allowed error ratio times the burn-rate threshold. For a 99.9% SLO, the fast-burn rule fires when the 1-hour and 5-minute error ratios are both above 1.44%.

What you need before you start

The examples below assume:

• http_requests_total counts eligible HTTP requests.
• The outcome label identifies requests as good or bad, using the same definition as the SLI.
• Each series has a service label.
• The SLO is 99.9% over 30 days, so the allowed error ratio is 0.001.

The thresholds come from the standard multi-window table. Multi-window burn-rate alerting explains how they are calculated.

Single PromQL example: the fast-burn condition

Written out in full, without recording rules, the fast-burn condition is:

(
  (
    sum(rate(http_requests_total{
      service="dispatch-api",
      outcome="bad"
    }[1h]))
    /
    sum(rate(http_requests_total{
      service="dispatch-api"
    }[1h]))
  ) > 0.0144
)
and
(
  (
    sum(rate(http_requests_total{
      service="dispatch-api",
      outcome="bad"
    }[5m]))
    /
    sum(rate(http_requests_total{
      service="dispatch-api"
    }[5m]))
  ) > 0.0144
)

The threshold 0.0144 represents a 1.44% bad-request ratio, equivalent to a 14.4× burn rate for a 99.9% SLO. The rule triggers when both observation windows exceed that threshold.

Recording rules for the error ratios

Calculating ratios over long windows for every alert is expensive. Recording rules compute each window once and keep the service label, so one set of alerts can cover many services:

groups:
  - name: slo-error-ratios
    rules:
      - record: service:error_ratio:rate5m
        expr: |
          sum by (service) (rate(http_requests_total{outcome="bad"}[5m]))
          /
          sum by (service) (rate(http_requests_total[5m]))
      - record: service:error_ratio:rate30m
        expr: |
          sum by (service) (rate(http_requests_total{outcome="bad"}[30m]))
          /
          sum by (service) (rate(http_requests_total[30m]))
      - record: service:error_ratio:rate1h
        expr: |
          sum by (service) (rate(http_requests_total{outcome="bad"}[1h]))
          /
          sum by (service) (rate(http_requests_total[1h]))
      - record: service:error_ratio:rate6h
        expr: |
          sum by (service) (rate(http_requests_total{outcome="bad"}[6h]))
          /
          sum by (service) (rate(http_requests_total[6h]))
      - record: service:error_ratio:rate3d
        expr: |
          sum by (service) (rate(http_requests_total{outcome="bad"}[3d]))
          /
          sum by (service) (rate(http_requests_total[3d]))

Alerting rules: fast, sustained and slow burn

groups:
  - name: slo-burn-rate-alerts
    rules:
      - alert: ErrorBudgetFastBurn
        expr: |
          service:error_ratio:rate1h{service="dispatch-api"} > (14.4 * 0.001)
          and
          service:error_ratio:rate5m{service="dispatch-api"} > (14.4 * 0.001)
        labels:
          severity: critical
        annotations:
          summary: "dispatch-api is burning error budget at over 14.4x"
      - alert: ErrorBudgetSustainedBurn
        expr: |
          service:error_ratio:rate6h{service="dispatch-api"} > (6 * 0.001)
          and
          service:error_ratio:rate30m{service="dispatch-api"} > (6 * 0.001)
        labels:
          severity: critical
        annotations:
          summary: "dispatch-api is burning error budget at over 6x"
      - alert: ErrorBudgetSlowBurn
        expr: |
          service:error_ratio:rate3d{service="dispatch-api"} > (1 * 0.001)
          and
          service:error_ratio:rate6h{service="dispatch-api"} > (1 * 0.001)
        labels:
          severity: ticket
        annotations:
          summary: "dispatch-api is using error budget faster than its SLO allows"

Writing the threshold as 14.4 * 0.001 keeps the burn rate and the SLO visible in the rule. For a different SLO, change 0.001 to that SLO’s allowed error ratio: 0.0005 for 99.95%, 0.0001 for 99.99%.

How to test burn-rate alerts

• Unit-test the rules with promtool test rules, feeding synthetic series that cross each threshold and checking the alert fires and clears.
• Replay a real incident: run the expressions over a past incident’s time range and confirm when each alert would have fired.
• Check the arithmetic: at a constant 14.4×, a 30-day budget loses 2% in one hour; confirm the fast-burn alert fires within that hour.
• Test the quiet case: a short spike that recovers in a minute should not fire anything.

Common mistakes with Prometheus SLO alerts

1. Dividing by zero at low traffic

With no requests in a short window, the ratio is undefined and the alert may flap or go silent. Consider a minimum request count, or a synthetic check, for low-traffic services.

2. Counting the wrong requests as bad

The outcome label must match the SLI exactly. Counting client errors or health checks distorts the burn rate.

3. Missing telemetry looks healthy

If the metric stops being scraped, the error ratio disappears rather than rising. Add an absent() alert for the request counter.

4. Inconsistent labels

If the good and bad series carry different labels, and will not match them. Aggregate with the same by (service) clause on both sides.

Frequently asked questions about Prometheus burn-rate alerts

Key takeaways

• A burn-rate alert compares error ratio against threshold × allowed error ratio in two windows.
• For 99.9%: 0.0144 over 1h and 5m, 0.006 over 6h and 30m, 0.001 over 3d and 6h.
• Recording rules keep long-window ratios cheap and reusable across services.
• Test with promtool, past incidents and the quiet case.