How to reduce alert fatigue on your on-call team

Why too many alerts make systems less reliable

When pages are frequent and mostly unactionable, people learn to ignore them, silence them, or sleep through them. The one alert that mattered then gets the same treatment as the nine that did not. Alert fatigue is not a discipline problem. It is a design problem, and it can be fixed by design.

A test for every page

A page should wake a person only if all of these are true:

• It is urgent: waiting until morning would cause real harm.
• It is actionable: a human can do something useful right now.
• It needs a human: automation cannot resolve it on its own.
• It signals user impact, or imminent impact, rather than just a suspicious internal number.

Start with an audit

Export the last month or two of alerts and sort them by how often they fired. For each frequent alert ask what happened as a result. If nobody did anything, remove it or demote it. If the same action was taken every time, automate it. If several alerts fired for one incident, merge them. The top ten noisiest alerts usually account for most of the noise.

Page on symptoms, ticket on causes

High CPU on one node is a cause that may or may not hurt anyone. A rising error rate or latency that users can feel is a symptom. Page on symptoms, and send cause-level signals to a ticket queue or a dashboard where they can be reviewed in working hours. Where you have SLOs, alerting on how fast the error budget is burning ties pages directly to user impact.

Use three tiers

• Page: urgent and actionable, goes to the on-call person.
• Ticket: needs attention soon but not now, goes to a queue.
• Dashboard only: useful context during an investigation, never notifies anyone.

Tune the rules

A few mechanical changes remove a surprising amount of noise. Require a condition to hold for a period before firing so that brief spikes do not page anyone. Group related alerts into a single notification and use inhibition so that a failed dependency suppresses the dozens of alerts it causes downstream. Silence known maintenance windows instead of letting them page.

groups:
  - name: api
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{job="api", code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "API error rate above 5% for 10 minutes"
          runbook_url: "https://runbooks.example.com/api-errors"

Give every alert an owner and a runbook

An alert without an owner will never be fixed, and an alert without a runbook forces the responder to rediscover the answer at 3 a.m. Link a short runbook from each page: what it means, how to confirm it, and the first things to try. Review alerts regularly, and delete the ones that no longer earn their place.

Measure the pager, then protect it

Track how many pages each on-call shift receives, how many were actionable, and how many fired outside working hours. Use those numbers in the same review where you discuss incidents, and treat a worsening trend as reliability work with the same standing as a feature. A quiet, trusted pager is one of the clearest signs of a healthy operation.