Learn/Alert Fatigue
CONCEPTS

Alert Fatigue

Desensitization caused by excessive or low-quality alerts, leading to missed critical alerts.

Alert Fatigue

Desensitization caused by excessive or low-quality alerts, leading to missed critical alerts.

The "Boy Who Cried Wolf"

Alert Fatigue occurs when engineers are exposed to frequent, non-actionable alerts. Over time, they become desensitized and start ignoring all alerts, including the critical ones.

The Psychology

The human brain ignores repetitive stimuli. If your phone beeps 50 times a day for "CPU High" and nothing bad happens, your brain learns that "Beep = Safety". When the "Database Down" beep happens, you ignore it.

Signs of Fatigue

  • Engineers ignoring pages until the 3rd notification.
  • "Oh, that's just the flaky backup job."
  • Sleeping through alerts.

The Cure

Delete your alerts.
Seriously. If an alert is not actionable (i.e., you don't do anything when you see it), delete it.
Only page for User Pain.

ExThe Target Breach

"Hackers infiltrated Target's network. The security system correctly detected the intrusion and fired an alert."

Impact
The security team, receiving thousands of alerts daily, assumed it was a false positive and ignored it.
Resolution
40 million credit cards were stolen. Result: Massive overhaul of alert thresholds to reduce noise.

Why Alert Fatigue Matters

Alert fatigue causes real incidents to be missed or ignored, increasing MTTR and customer impact.

According to State of IM 2025, teams with high alert fatigue have 67% longer MTTR.

Common Pitfalls

Alerting on "Warning" states
Don't page for "Disk 80% Full". Page for "Disk will be full in 4 hours". Better yet, auto-scale the disk and don't page at all.

How to Use Alert Fatigue

🎯
Alert on Symptoms: Alert on user impact, not metrics.
📊
Track Alert Rate: Measure alerts per engineer per week.
🔇
Noisy? Kill It: If it doesn't wake you up, don't page.

Frequently Asked Questions

How many alerts is too many?
More than 1-2 pages per shift (12 hours) creates fatigue. Ideally, on-call should be silent unless things are actually broken.

Learn More