Retries¶
Why a failed backup is not an incident¶
A device unreachable at two in the morning is rarely unreachable at three. A reboot, a maintenance window, a saturated link: most failures repair themselves before anyone reads about them.
An operator who opens the morning report and finds a page of exceptions that have already resolved learns, within a week, to stop opening the report. That is the worst outcome a report can produce — worse than no report at all, because it looks like supervision.
So a failed backup is replayed on its own, spaced further apart each time, and only the last attempt counts as an incident. Three attempts of the same backup are one failure, not three: they stay linked as one series, and the reports count them once.
The cascade¶
A retry strategy is set at three levels, exactly like retention:
The most specific value that is set wins, field by field. A customer that only sets the number of attempts leaves the delays inherited from the global level. An empty field is not zero — it means “as above”.
Each screen shows where every value comes from, so that “why was this device only retried once?” is answered without opening three others.
The fields of a rule¶
| Field | Screen | Effect |
|---|---|---|
attempts |
Attempts | total number of tries, the first one included |
delay_seconds |
First delay | seconds between the first failure and the second try |
backoff |
Spacing factor | each delay is multiplied by this factor |
max_delay_seconds |
Delay ceiling | a delay never grows beyond this |
Note the first line: attempts = 3 means one try and two retries, not four
passes. It is the number an operator counts in the run history.
What the schedule actually looks like¶
The three delay fields are easier to judge as a timetable than as parameters. With the default rule — first delay 300 s, spacing factor 3.0, ceiling 3600 s — here is what a device actually does.
With 3 attempts (the default):
| Attempt | Waits | Since the scheduled time |
|---|---|---|
| 1 | — | 0 |
| 2 | 5 min | 5 min |
| 3 | 15 min | 20 min |
The failure is reported 20 minutes after the scheduled time, at the latest.
With 5 attempts:
| Attempt | Waits | Since the scheduled time |
|---|---|---|
| 1 | — | 0 |
| 2 | 5 min | 5 min |
| 3 | 15 min | 20 min |
| 4 | 45 min | 1 h 05 |
| 5 | 60 min | 2 h 05 |
Two attempts more than double the window, from 20 minutes to a little over two hours. That is the point of the spacing factor: a single setting covers both a device rebooting (minutes) and a maintenance window (hours), without hammering the device in between.
Why there is a ceiling¶
The fifth attempt above should have waited 2700 × 3 = 8100 s, or two hours and fifteen minutes. The ceiling caps it at 3600 s.
Without that cap, an exponential sequence leaves the day it started. The eighth attempt of a generous rule would land three days later, against a device whose configuration has changed twice since — and the backup, if it finally succeeded, would be dated at a moment nobody would think to look for. The ceiling keeps a series inside a horizon an operator can still reason about.
Choosing a rule¶
Two questions decide it, and neither is “how many retries do I want”.
How long is a normal outage on this fleet? The retry window should cover it. A campus of switches that reboot for firmware updates needs the two-hour window; a data-centre firewall that is either up or genuinely broken does not — the extra attempts only delay the alert.
How much does an alert cost? The window is also the delay before anyone is told. A fleet monitored by an on-call rota pays for every minute of it. Widening the retry window buys fewer false alerts and slower true ones; there is no setting that gives both.
The default — three attempts over twenty minutes — is the compromise for a fleet backed up nightly. A device backed up every fifteen minutes should not be given a two-hour retry window: the next scheduled run would overtake it.
When you are told¶
Nothing is sent while a retry is still pending: an alert that arrives before the repair teaches people to ignore alerts. The notification goes out once the series is exhausted, and carries the reason, the transport used and the number of attempts.
Alert destinations — email addresses and webhooks — are configured per customer and for the whole platform, on the customer page and in the settings.