Your pipeline deleted its own alarm (two greps to check)

A reader told me my fix was only a promise

Count the scheduled jobs in your repository that end by sending something. A report, a digest, a backup confirmation, an alert. Now say, without opening anything, which of them actually sent something yesterday. Most teams cannot answer that, and the cost stays at zero until the one day it does not.

I wrote about this gap two days ago and shipped a guard for it: any script whose own header says it runs daily must appear in a workflow file that has a schedule. A reader named Mads Hansen replied that this does not go far enough. A guard that reads workflow files proves only that a declaration exists in source control. He suggested a deployed canary instead: run the thing on the real path, and check it uses the same identity, secrets and result sink as production.

He was right within a day, and it cost me two days of silence.

The change was green on day one and broke on day two

I built a second daily probe and, on his advice, ran it once by hand against production instead of waiting for its first scheduled run. It went red immediately. Then it went red again for a different reason. The second failure was not in the new probe at all. It was in the older one that had been running for weeks.

Here is the mechanism, and it is worth borrowing. The probe writes its history to a separate branch, so the main branch does not collect one commit per data point. The step checks that branch out, appends a line, pushes. That branch contains exactly one file. Checking it out therefore removes every other file from the working tree, including the notifier that the next step calls.

On the day I wrote that step, the branch did not exist yet. So the code took the other path, the one that creates an orphan branch, and that path leaves the working tree alone. The run was green. The next morning the branch existed, the first path ran, and the notifier was gone. The daily message stopped arriving, and the only trace was a red run nobody read.

A change that passes on the day it ships and fails the day after is the worst kind, because the review, the test and your memory of it are all from day one.

Ten minutes, three checks, no tooling required

Steal this. Two greps and one command, and the first two work on any repository with GitHub Actions in it.

# 1. Scheduled workflows that also push to a branch.
grep -l 'schedule:' .github/workflows/*.yml | xargs grep -l 'git push'

# 2. In each hit: does any step AFTER that push run a script from the
#    repo? The branch checkout may have removed it. This is a handful
#    of files - read them, do not automate it.

# 3. The check no file in your repo can do for you:
gh workflow run <name>.yml && gh run watch

If step two finds anything, the fix is to move the notification before the push. The message matters more than the bookkeeping, and a failed push must never be allowed to silence it.

Then do the one that has nothing to do with branches. Trigger one scheduled job by hand today, against production, and watch it. Not the test suite. The job. Everything you learn in that minute is something no file in your repository could have told you.

Make sure the alarm can still say no

A canary that always passes is just another thing to maintain. So break it on purpose once. Remove the token, point the notifier at a wrong channel, and confirm that the run goes red and that you hear about it.

The new probe aborts when its API key is missing, before it touches the network, and it exits non-zero. It does not print a tidy empty report. That distinction is the whole point: found nothing and never asked must not look the same, and if your job prints the same thing in both cases, it is lying to you politely.

The same shape, three more places

Any step that switches branches or checks something out mid-job. Everything after it is running against a working tree you did not think about.

Any create-if-missing path. The first run takes one branch of the code and every later run takes the other, so the version that matters was never the version you reviewed.

Any notifier called through plain curl. curl exits 0 as soon as the request was made, including a 401 where nothing was delivered. Green step, no message, no trace.

What changed, and what is still a promise

The probe now runs on the real path, and I watched it fail twice before it worked: wrong permissions on the token, then the missing notifier. Both took under a minute to find, because they happened when I asked instead of at 07:00 on some morning in two weeks. The older probe sends its daily message again, and I only know that because I made it send one while watching.

Mads suggested something else that I have not built: a durable ledger, a row written before each run is due, so that ran and found nothing is mechanically different from never ran. My canary proves the path works today. A ledger would prove it ran yesterday. That part is still a promise in source control, which is exactly what he warned me about, and I would rather write that down than pretend otherwise.