The Alarm Nobody Heard
Monitoring that records problems but never tells a person is decoration. It looks like protection on a dashboard, and it does nothing when something breaks. If you haven't watched one of your alerts reach a human being recently, assume it doesn't. We learned this on our own systems, and the client found the problem before our monitoring did. Here's what happened, what we changed, and how to check whether you have monitoring alerts nobody sees.
What happened on a system we run
We run a production integration for a client. After a software update, that integration started skipping an important write. The work it was supposed to record simply didn't get recorded.
The error showed up in several of our systems within minutes. That part worked. The cause of the error only reached the server's console output, the scrolling text a developer reads when they're already looking at the server. Nobody was looking.
Our alert collector saw the new error type within minutes. Then it dropped it. The collector worked from a list of known-bad error types, and this one matched nothing on the list. It was new, so it was ignored.
A spike alert did fire. It went into storage and stopped there. Our alerting had been running in observe-only mode for weeks, recording every alert it would have sent and sending none of them.
Then the client told us something was wrong. Several layers we'd built to catch this caught a piece of it, and not one of them told a person.
Why each layer failed
We want to be precise about this, because the failure was ordinary and the pieces are common.
The cause stayed in one place. An error that says "failed" without saying why forces someone to go dig. When the why lives only in console output, it's gone the moment that output scrolls away.
The collector only knew the past. A list of known-bad patterns catches the failures you already predicted. A software update creates failures you didn't predict. That's exactly the case a known-bad list can't cover.
Observe mode never got switched off. Observe-only mode is a reasonable way to tune alerts before they start waking people up. Left on for weeks, it becomes a filing cabinet for emergencies.
None of these was dramatic. Each one was a setting or a design choice that made sense when someone made it. Together they produced a system that knew about the problem and kept it to itself.
We've written before about monthly reports clients glance at and move on from. An alert that lands in storage is the same problem with higher stakes. Information that never reaches a decision-maker hasn't done its job.
What we changed
The collector flags anything new. We replaced the known-bad list. The collector now flags any error type it hasn't seen before, with a short list of deliberate exclusions per app. The default is to speak up. Silence has to be chosen on purpose and written down.
Alerts carry the cause. An alert now says what broke and why. The person receiving it can start fixing instead of starting an investigation.
A logging rule for every important action. We adopted a rule across our systems. Every stored error carries its cause. Every place the software skips or gives up on an important outbound action leaves one record saying why.
Alerting sends. We switched alerting from observe-only to sending. The very first check after the switch delivered alerts that observe mode had been storing silently: items in four work queues that had been waiting far longer than their limits. We just hadn't been hearing them.
That's the part we'd ask every business owner to sit with. The moment we turned the sound on, the system had a backlog of things to tell us.
How to check your own monitoring
This applies to any business that depends on software to take orders, process work, or move data. You don't need to be technical to run these checks. You need someone who can answer the questions honestly.
1. Fire every alert once, on purpose
Trigger each alert deliberately and confirm that a named person received it. Get a name, not a channel or a team inbox. If you can't make an alert fire on demand, you don't know that it works. Do this after any software update too, since an update is exactly when things change underneath you.
2. Look for test, preview, or observe-only modes
Ask whoever manages your tools whether anything is running in a test, preview, sandbox, or observe-only mode. These settings are useful during setup and easy to forget afterward. Ours ran for weeks. Check every tool that's supposed to warn you about something, including the ones nobody has opened in a while.
3. Alert on the unexpected
If your monitoring only watches for errors someone listed in advance, it'll miss the new one. Ask for alerts on any error type the system hasn't seen before. Keep exclusions short, and write down why each one exists. You'll get more noise at first. That noise is the system finally telling you what it sees.
4. Make each alert say what broke and why
Read your last few alerts. Could the person receiving them act without logging in somewhere else to find out what happened? If an alert only says "error" or "failed," the cause is sitting somewhere nobody's looking. Push the cause into the alert itself.
5. Know who receives each alert and what they do next
Every alert needs an owner and an expected response. Write it down: who gets this, what they check first, and who they call if they can't fix it. An alert that goes to everyone tends to get handled by no one. An alert that goes to someone who left the company goes nowhere at all.
6. Read the backlog before you switch on
If you find an alert system that's been quietly recording, look at what it recorded before you flip it to sending. That backlog is real work. When we switched ours on, the first check surfaced items in four queues waiting far past their limits. Plan for that. Have someone ready to work through the list, and don't mute it because it's loud on day one.
Where we could be wrong
There's a fair objection. Alerting on every new error type will generate more alerts, and too many alerts teach people to ignore them. That's true, and it's why we keep deliberate exclusions. But we'd still choose a noisy alarm we can tune over a quiet one that misses the failure we didn't predict. If your team is already drowning in alerts, fix routing and ownership first (checks 4 and 5), then widen what you watch.
Why we're telling you this
We run our own systems the way we ask clients to run theirs, and sometimes that means publishing what went wrong. We've made the case before that reliable work shows its proof instead of describing itself. This is that proof, including the part where the client beat our monitoring to the news.
The fix wasn't exotic. We flagged the unknown, put the cause in the alert, adopted a rule to log every skipped action with a reason, and turned the sound on. Then we read what the system had been trying to tell us.
Find out which of your alarms reach a person
At Metrix Digital, we review monitoring and alerting setups for businesses that depend on software to run. We'll trace each alert from the moment it fires to the person it's supposed to reach, and we'll tell you plainly which ones arrive, which ones stop in storage, and which ones would miss a failure nobody predicted.
Get an audit. Contact us, and we'll show you which of your alarms someone actually hears.