Collector errors
When a check cannot run, it reports an error kind instead of a value. The kind is shown on the check, in the resource's timeline and in the result of Test connection, and it narrows the cause. This article lists every kind, explains the Cannot reach alert, and says what to check for each symptom, for checks and for sources.
Error kinds
| Kind | Shown as | Means | Counts towards Cannot reach |
|---|---|---|---|
unreachable | Unreachable | Network error, name not resolved, connection refused or dropped, TLS handshake failure | Yes |
auth | Sign-in refused | Login failed, or the login cannot open the database it needs | Yes |
timeout | Timed out | Connecting, or the whole check, took longer than allowed | Yes |
query | Query failed | The system answered, but the statement failed, usually for a missing permission | No |
query_timeout | Query timed out | Connected, but the statement exceeded its timeout | No |
config | Configuration | Something to fix in Excubius: no host, no credential or the wrong kind, a secret that cannot be read | No |
unsupported | Not supported | The collector has no transport for this check | No |
internal | Collector error | A bug in the collector; the message says where | No |
A resource shows a single "Cannot reach" alert
Symptom. One critical alert Cannot reach appears on a resource, and its other alerts list it as their root cause.
Cause. By default, 3 consecutive runs of a check of that resource failed with unreachable, auth or timeout. Instead of one alert per failing check, Excubius opens one alert and holds the notifications of the resource's other alerts while it is open. The number of failures is EXCUBIUS_UNREACHABLE_AFTER.
Resolution. Fix the connection, then select Test connection on the resource to confirm. The alert resolves by itself as soon as the failing checks succeed again; until then, the notifications of the resource's other alerts wait.
Checks fail with auth
Cause. The password is wrong or expired, the login is disabled, the instance does not accept SQL logins, or the login has no user in a database a check opens. On an availability group, the login may have a different SID on the new primary after a failover.
Resolution. Check the credential and that the instance runs in mixed mode. On availability groups, create the login with the same SID on every replica; see Prerequisites for availability groups.
Checks fail with unreachable or timeout
Cause. The host name does not resolve from the Excubius server, a firewall blocks the port, the service is down, or the network path is slow.
Resolution. Check DNS and the firewall between the Excubius server and the system, on the right port: the SQL port (and UDP 1434 for named instances without a fixed port), 443 or 80 for web endpoints, 5986 for WinRM, 22 for SSH. For multi-subnet listeners, allow every listener IP. See Network paths.
Checks fail with query
Cause. The system answered, but the statement was refused, almost always because the login lacks a permission. A check that needs msdb or CONNECT ANY DATABASE fails this way while the others work.
Resolution. Select Test connection: on SQL Server it lists the permissions of the login and what is missing. Grant what Permissions lists, and nothing more.
Checks fail with query_timeout
Cause. The statement ran longer than its timeout, 30 seconds by default on SQL Server, usually on an instance under heavy load.
Resolution. Try again later. If it persists on a busy instance, raise the check's query_timeout_seconds parameter for that resource.
Checks fail with config
Cause. Something in Excubius is incomplete: the resource has no host, no credential reaches it, the credential is of a kind the check cannot use, or the stored secret cannot be decrypted because EXCUBIUS_DATA_KEY changed.
Resolution. Edit the resource or the credential. A check under a report server that holds a Windows account needs a SQL login for its T-SQL checks; see Which credential a resource uses. If secrets are unreadable, enter them again.
Checks fail with unsupported, or stay Unknown
Cause. No collector offers the transport the check needs. EXCUBIUS_COLLECTOR_TRANSPORTS may leave it out, or the transport does not exist yet: PostgreSQL checks stay Unknown in this release.
Resolution. Remove the transport from the restriction, or switch the check off on that resource. See Compatibility.
A job ends with "No collector picked it up"
Symptom. Test connection, Discover or a source's Test stays at waiting for the collector, then ends with No collector picked it up.
Cause. No collector claimed the job within 10 minutes. The collector is not running, has no token (EXCUBIUS_COLLECTOR_TOKEN is empty, and it then idles), or does not offer the transport. A job that a collector claimed but did not report back within 15 minutes fails with timeout.
Resolution. Check the collector's last heartbeat in the Sources zone of the Overview. On the server, as the service account, run <dir>/current/scripts/compose.sh ps and <dir>/current/scripts/compose.sh logs collector.
Source errors
A source shows its last error with one of these kinds:
| Shown as | Usually | Check |
|---|---|---|
| Credentials refused | Wrong or expired key or secret, or the wrong region | Replace the secret, check the region or data centre |
| Missing permission | A role or scope is missing | Test says which read failed; grant what Permissions lists |
| Not found | Wrong account ID, organization or subscription | Check the IDs on the source |
| Rate limited | The platform throttles the requests | Raise the polling intervals |
| Unreachable, Timed out | The server cannot reach the platform's API | Allow outbound HTTPS to the platform |
| Platform error | The platform's API answered with an error | Try again later; check the platform's status |
| Secrets unreadable | EXCUBIUS_DATA_KEY changed since the secrets were saved | Enter them again |
| No collector picked it up | The collector is not running | See the previous section |
| Freshness Stale | No successful poll in three intervals, at least 15 minutes; for a webhook-only source, nothing received for a day | Is the collector running? Is the platform sending webhooks? |
| Webhooks Rejected | The source is switched off or in Polling mode, or the secret header is wrong | Check the mode and the header the platform sends |