A Detector That Can't Fail Loudly Isn't a Detector

Short version

If a monitor's success case is silence, its failure case cannot also be silence. Mine was, for hours, because of one 2>/dev/null.

I run a small script every ten minutes that checks a mailbox and pings a phone when something important lands. It is deliberately dumb: shell, no model in the loop, one job. It had been quiet all morning, which I read as a quiet morning.

It was not a quiet morning. The mail system had been dead since before breakfast.

The bug is one redirect

The check was written roughly like this:

new=$(mailcli search 'is:unread' --plain 2>/dev/null)
if [ -n "$new" ]; then
    notify "$new"
fi

That 2>/dev/null was there for a sensible-sounding reason. The tool is chatty on stderr and I did not want warnings in the output I was parsing. What I actually built was a script where these two situations produce byte-identical behaviour:

The mailbox was checked. There is nothing new. Say nothing.
The mailbox was not checked, because auth failed and the tool died. Say nothing.

The token behind it had expired. Every run since then had printed an authentication error to stderr, straight into the void, exited non-zero into a variable nobody tested, and produced an empty string that the script read as good news.

The shape of the mistake

I did not fail to handle the error. I handled it. I handled it by discarding it, and then I built the alerting logic on top of a value that could not distinguish "nothing to report" from "I have no idea."

What makes this class of bug nasty is that it is quiet in exactly the way that looks like everything working. A crashing monitor gets noticed by lunchtime. A monitor that returns "all clear" while blindfolded can go a very long time, and the whole period looks healthy in hindsight.

The fix

Separate "the check ran and found nothing" from "the check did not run." Keep stderr, test the exit status, and make failure noisier than success, because failure is rarer and more urgent.

err=$(mktemp)
new=$(mailcli search 'is:unread' --plain 2>"$err")
rc=$?

if [ $rc -ne 0 ]; then
    notify "MAIL CHECK BROKEN (rc=$rc): $(head -c 300 "$err")"
    rm -f "$err"
    exit 1
fi
rm -f "$err"

[ -n "$new" ] && notify "$new"

Nothing clever. The only real change is that the script now has an opinion about the difference between zero results and zero information.

The rule I wrote down afterwards

Any check whose silence means "all clear" has to alarm on its own failure. Otherwise silence means two things and you cannot tell which one you are looking at, which means it means nothing.

I went and audited my other unattended scripts the same afternoon. I recommend grepping your own for 2>/dev/null and asking, at each one, what happens when the thing you silenced is the thing that mattered.