Alerts & escalation

An alert is what happens when a monitor crosses from late to down. This page is about the three knobs that decide when that happens and how often you hear about it afterwards.

Before the first alert

Two settings sit between “a ping is late” and “you get told”.

Grace is the lateness you consider normal — covered in Concepts. Nothing alerts inside it.

Consecutive misses is the second filter. A monitor can be told to alert only after n deadlines have passed unmet, rather than the first. For a job that runs every minute and occasionally skips one, requiring two or three misses removes a whole class of noise without meaningfully delaying the signal — three missed minutes is still three minutes.

For a job that runs daily, leave it at one. The whole point is that you find out today, not in three days.

Escalation

The first alert says the job is down. Escalation decides what happens if it stays down.

Escalation is a list of intervals measured from the moment the monitor went down. Each one fires a follow-up:

Configuration	Behaviour
(empty)	One alert. Nothing further.
1h	The initial alert, then one reminder an hour after it went down.
1h, 4h	Reminder at one hour, another at four.

The initial alert is implicit and is never part of the list — the intervals are the schedule for follow-ups on top of it.

Repeat the last interval

There’s a switch to keep nagging at the final interval indefinitely instead of stopping when the list runs out. With 1h, 4h and repeat enabled, you get reminders at one hour, four hours, and then every four hours until the monitor recovers.

This is “scream until acknowledged”, and it’s off by default. Bounded escalation is the right default because an unbounded reminder loop trains people to mute the channel, and a muted channel is worse than no escalation at all. Turn it on for the jobs where silence genuinely is not an acceptable outcome, and leave it off everywhere else.

Caution

Escalation multiplies with routing. A monitor whose alerts reach three channels sends every follow-up to all three. Enabling repeat-last on a monitor routed to a wide workspace-level channel is how a single stuck job becomes a hundred messages overnight.

Recovery

When a down monitor checks in again, a recovery event fires. It is a separate routing event from down, so a channel subscribed only to down will report every failure and never report a fix.

Subscribe to both unless you have a specific reason not to — see Integrations.

Quiet hours interact with all of this

If a monitor has quiet hours configured, alerts and escalations generated inside that window are deferred to the end of it, not dropped. A job that goes down at 02:00 with a 22:00–08:00 window produces its alert at 08:00.

That means escalation intervals do not fire on their own schedule during quiet hours — you get the outcome when the window ends rather than a queue of overnight reminders.

Reading the history

Every alert is recorded: what fired, when, which channels it went to, and whether delivery succeeded. That last part matters more than it sounds — a Slack webhook revoked three weeks ago produces alerts that look sent and arrive nowhere, and the delivery record is the only place that’s visible.

Alongside it, each period a monitor spent down is recorded as an outage you can annotate.

The two answer different questions. The alert history answers “were we told?” The outage history answers “what actually happened?”