Alerts and notifications
An alert in Excubius stands for one condition on one resource, from the moment it is detected until it clears. Excubius keeps one alert per condition however often it fires, groups alerts under a root cause when one failure explains others, and sends a message only when there is something new to say. This article explains the life of an alert and the rules that decide whether, when and where a notification goes.
From a result to a message
Monitoring never waits for Slack: results are stored and evaluated first, and messages go out right after and from a background sweep every 15 seconds.
The life of an alert
| Status | Means |
|---|---|
| Open | The condition holds and nobody has acknowledged it |
| Acknowledged | Someone has it. Their name and the time are recorded |
| Silenced | Notifications wait until a time someone chose, from 1 minute to 30 days. When the time is up the alert is Open again |
| Resolved | The condition is over. Resolved alerts take no more actions and stay in the history |
An alert resolves on its own when the check recovers, when the source closes it, when the thing it measured disappears (a database that was dropped, for example) or, for a Cannot reach alert, when the resource answers again. An operator can also resolve it by hand with a note; if the condition still holds, the next check run opens a new alert. When a Jira ticket for an event alert, such as a failover, reaches a done status, the alert resolves too.
The inbox also holds security findings, which can be accepted as a risk, and advisories from the updates module. They follow the same life, with the differences explained in Security posture.
One alert per condition
Excubius identifies a condition by three things: the resource, the check and the check's labels, such as the database name or the wait type. Together they form the alert's fingerprint. While the condition holds, every new result updates the same alert: its value follows the latest run and its counter of occurrences grows. The title is the check name followed by the label values, for example Last full backup age — Sales.
When the condition clears and later comes back, a new alert opens with the same fingerprint, so the history keeps every episode. Alerts that come from a source, such as a New Relic issue or an Azure Monitor alert rule, are kept as one alert per vendor rule and resource, however many times the vendor raises it.
Escalation and reminders
- When a warning becomes critical, the alert escalates: the change is recorded on its timeline and the channel is told at once.
- When a critical alert eases to a warning, the change is recorded but not sent.
- Expiry checks remind you before the deadline: a warning 30 days before, a reminder at 14 days and critical at 7 days.
Escalation by time, such as notifying another channel when nobody acknowledges a critical alert within 15 minutes, is not available yet. Today an alert reaches the channels of its entity once, plus escalations, reminders and its resolution.
Root cause: Cannot reach
When the collector fails to connect to a resource three times in a row because it is unreachable, refuses the sign-in or times out, Excubius opens one critical Cannot reach alert for that resource. The alert lists the checks that cannot connect and how many failures each has had.
Every other alert of the resource, open already or opened while the resource is unreachable, becomes a dependent of that alert. The detail panel of a dependent shows Caused by, and the root cause shows how many alerts depend on it. Dependents are not notified while the root cause is open, so the team gets one message instead of one per check. The root cause resolves on its own when no check fails to connect any more.
When a message is sent
| Moment | Message |
|---|---|
| An alert opens | Yes, to every channel whose minimum severity it reaches |
| Its severity rises | Yes, even during the quiet period |
| A reminder day of an expiry check | Yes |
| It resolves | Only if its opening was announced |
| It repeats, eases or is acknowledged | No; the change is recorded on the timeline |
Each message carries the resource and its entity path, the check, the value against the threshold and where that threshold comes from, since when, the suggested action from the catalog and a link to the alert. Messages never name the product, so an installation that uses its own brand keeps it.
Quiet period. Excubius sends at most one message per condition every 10 minutes, unless the severity rises. A condition that resolves and comes back within that time is not announced twice. If a held-back message still applies when the period ends, a later sweep sends it. An administrator can change the period for the whole server.
Failed deliveries. A message Slack refuses is recorded on the alert's timeline and retried by the next sweeps, three attempts in all.
Silences and maintenance windows
Both keep a channel quiet. They differ in what they cover and how they start.
| Silence | Maintenance window | |
|---|---|---|
| Covers | One existing alert | A whole entity, or some resources and what was discovered under them, including alerts that do not exist yet |
| Who starts it | An operator, from the alert | An operator, from Settings › Maintenance windows |
| How long | 1 hour, 4 hours, 1 day, or 1 minute to 30 days | Now, for a set time; once; or every week |
| During it | Nothing is sent for the alert but its resolution | Nothing is sent about the covered resources. They show Maintenance |
| When it ends | The alert is Open again, and anything the channel has not heard yet, such as an escalation, goes out | Alerts still open are announced. Alerts that opened and cleared inside the window are never announced |
In both cases checks keep running and history stays complete. Use a silence when you know the cause of one alert and need quiet while you fix it. Use a maintenance window for planned work.
How a message finds its channel
Notification channels are Slack channels set on entities. An alert goes to the enabled channels of the nearest entity that has any, walking up the tree from the alert's entity. An entity inherits its ancestors' channels until it defines its own.
- Each channel has a minimum severity, Info, Warning or Critical, and a message language, English or Spanish when the installation offers languages. Alerts below the minimum are not posted there.
- When no entity on the path has a channel, the server's default channel, if the administrator set one, receives warnings and criticals.
- Settings › Notifications shows, for any entity, Where alerts go: its own channels or the inherited ones.