Alert fatigue kills incident response. Here's how to build an alerting strategy that fires when it matters, stays quiet when it doesn't, and actually gets fixed when it goes off.
You set up an alert. It fires at 2am. You check — nothing's actually broken. You silence it. A week later it fires again. You add it to your mental "ignore" list. Three months later, the thing it was supposed to catch actually breaks, the alert fires, and you don't notice because you trained yourself to ignore it.
This is alert fatigue. It's not a configuration problem — it's a design problem.
If an alert fires and your response is "check if it's real," the alert is wrong. Alerts should only fire when a human needs to do something specific. If the answer is "wait and see" or "it'll probably resolve itself," you don't need an alert — you need a dashboard.
This sounds obvious. It's almost universally violated.
It describes a user-visible problem, not a metric threshold. "Error rate above 2%" is a metric. "Users are seeing errors on the checkout page" is a problem. Write alerts that map to the second kind, even if you implement them using the first.
It fires rarely enough that you trust it. An alert that fires multiple times per week trains your team to ignore it. Aim for fewer than 5 alert firings per service per week during normal operations. If you're above that, tune it.
It includes enough context to act immediately. When an alert fires, the responder should know: what broke, which service, when it started, and what the impact is. Don't just say "error rate high" — include the error rate, the baseline, and a link to the logs.
It has a clear owner. Alerts sent to a general #alerts Slack channel that nobody owns are alerts nobody responds to. Route alerts to the team responsible for the service.
Best for: detecting user-facing failures fast.
Condition: error_rate > 5% over last 5 minutes
Cooldown: 30 minutes
Channel: Slack #oncall
The threshold depends on your baseline. If your service normally runs at 0.1% errors, 2% is a fire. If it normally runs at 1%, 5% might be normal traffic variation. Establish a baseline first — LogFlow's anomaly detection does this automatically by comparing current error rate against a rolling 7-day baseline.
Best for: detecting silent failures (service stopped logging = service stopped running).
Condition: log_count < 10 over last 10 minutes
Cooldown: 60 minutes
A service that suddenly goes quiet is often more alarming than one with elevated errors. Silence detection is one of the four anomaly types LogFlow watches for automatically.
Best for: critical, named errors that should never appear in production.
Condition: message contains "FATAL" OR message contains "database connection refused"
Cooldown: 5 minutes
These are your "something is specifically wrong" alerts. Keep this list short — 5 to 10 specific strings that should never appear in production. If you're adding dozens of keywords, you've shifted from alerting to log monitoring, which is a different tool (your Logs Explorer).
Best for: catching sudden shifts in log severity distribution.
If your service normally logs 95% info and 2% error, a sudden shift to 20% error is meaningful even if the absolute error rate is low. LogFlow's level_spike condition tracks this ratio.
In your Alerts dashboard, create a new alert:
The cooldown is the most underused setting. A 30-minute cooldown means if the condition stays true, you get one alert, not thirty. You investigate once. If it hasn't resolved after 30 minutes, it fires again as a reminder.
Most teams set cooldowns too short. An alert that fires every 5 minutes for a 2-hour incident sends 24 notifications. After the third one, responders stop reading them.
A better model: fire fast, then go quiet. Alert immediately when the condition starts. Fire again once per hour if it's still ongoing. This gives you immediate awareness without drowning your team.
| Alert Type | Initial Cooldown | Ongoing Cooldown |
|---|---|---|
| P0 (outage) | 5 min | 30 min |
| P1 (degraded) | 15 min | 60 min |
| P2 (elevated errors) | 30 min | 4 hours |
| Info (watch) | 60 min | Don't alert |
Manual thresholds (error_rate > 5%) are static. They don't adapt to your traffic patterns. A 5% error rate at 3am (10 requests/minute) is very different from 5% at 3pm (10,000 requests/minute).
LogFlow's anomaly detection adjusts automatically. It looks at historical patterns for that time of day and day of week, then alerts when current behavior deviates significantly. You don't set a threshold — you let the system learn what normal looks like.
Use static thresholds for things that are always wrong regardless of traffic (a FATAL log, a specific error code). Use anomaly detection for things that are wrong relative to baseline (error rate, log volume).
Use different channels for different severity levels:
P0 (outage) → PagerDuty or phone call
P1 (degraded) → Slack DM to oncall
P2 (elevated) → Slack #alerts channel
Anomaly (watch) → Email digest
LogFlow supports Slack, email, Telegram, Discord, and webhooks. Wire Slack for fast response, email for summaries, and webhooks to integrate with your incident management system.
Once a month, look at your alert history. For each alert that fired:
Alerts that regularly resolve without intervention should be demoted to dashboards. Alerts that are always real should have their threshold tightened. Alerts that never fire need to be questioned — either they're never triggered (good?) or the threshold is too high.
If you're setting up alerting for the first time, start with three alerts:
That's it. Run with those for a month. You'll learn quickly which ones are too noisy, which thresholds are wrong, and what you actually need to add.
Don't try to alert on everything. Alert on the things that require a human decision right now. Everything else is a metric for a dashboard.
Related: Debugging Microservices with Distributed Tracing · Structured Logging Guide · Set Up Slack Alerts Tutorial
Free plan available. No credit card required. Up and running in 2 minutes.
Get started freeLog Retention: How Long Should You Actually Keep Logs?
Most teams keep logs too long or not long enough. Here's how to set a retention policy that actually makes sense.
Structured Logging: Stop Grep-ing Through Plain Text
Plain text logs break the moment you need to filter or alert. Structured logging with JSON gives you queryable logs from day one.
You Don't Need Datadog: Log Management for Startups That Ship Fast
Enterprise observability platforms charge enterprise prices. Here's what startups actually need — and why most are overpaying by 10x.