Change thresholds and turn checks off
Every check starts from the baseline of its catalog entry. You change it where it makes sense: on an entity, so that every resource below it follows, or on a single resource that is the exception. This guide shows both, how to go back to the inherited values, and how to turn a check or one of its levels off.
Before you begin
- To change a check on one resource, you need the Editor role or higher on the resource's entity.
- To set an entity default, you need the Entity admin role on that entity.
- Know where values come from: baseline, then each entity from the root down, then the resource. The most specific value wins. See Thresholds and baselines.
Set values as high in the tree as you can and override only the exceptions. A default set once on Ciguapa Inc. › Production covers every server added there later.
How a level is written
Each check has a Warning level and a Critical level. Each level has:
| Part | Values |
|---|---|
| Operator | >, >=, <, <=, ==, !=, in, not_in, matches |
| Value | A number in the check's unit, shown next to the field, or a list for in and not_in |
| For | Optional. How long the condition must hold before the alert opens, such as 30s, 15m, 1h or 1d. It absorbs short spikes; recovery is immediate |
Some checks compare states rather than a value against limits, for example the health of an availability group replica. Their rules come from the catalog, and the editor says so instead of showing levels.
Change a threshold on one resource
- Open the resource and select the Checks tab. Use Search checks or Only problems to find the check.
- Select the pencil icon, Edit, on the check's row. The dialog shows the check's id, its baseline and the current levels.
- For each level you want to change, keep This level raises alerts selected and enter the Operator, the Value and, if needed, For. For example, for Page life expectancy (
sqlserver.ple_seconds) onsql-ventas-01, set Warning level to<900for10m. - Select Save.
The row now shows the new levels marked (this resource). Levels you did not change keep their inherited values.
To go back to the inherited values, open the dialog again and select Back to inherited thresholds. The button appears only when the resource has values of its own.
Set a default for an entity
- Open Settings › Entities.
- Open the actions menu of the entity, for example Production, and select Threshold defaults. The panel lists the checks this entity sets itself; it inherits every other one.
- Under Set a default for a check, choose the check, for example CPU utilization (SQL process) (
sqlserver.cpu_percent), and select Add. - For each level, choose:
- Inherit: keep what comes from above.
- Set here: enter the Operator, Value and For, for example
>75for15m. - Turn off here: this level raises no alerts on this entity and below.
- Select Save.
Every resource below the entity now uses the default, unless a lower entity or the resource changes it. On each resource's Checks tab, the level shows the entity it comes from.
To remove a default, select the trash icon next to it, Remove the default of that check. Resources below then inherit from further up again.
Turn a check off
On one resource, open the Checks tab and switch the check off in the On column, or clear Check runs in the check's dialog. The check stops running on that resource and is listed as Off, so nobody forgets it exists. Switch it on again the same way.
To stop a check from alerting on a whole entity, open its Threshold defaults, add the check, and set both Warning level and Critical level to Turn off here. The check keeps running and recording values on every resource below, but it raises no alerts there. Use it for checks that make no sense in a branch, such as report server checks where there is no report server.
Change how often a check runs
In the check's dialog, enter an Interval from 10s to 7d, for example 30s, 5m or 1h. Leave it empty to use the catalog's interval, which the field shows. The row marks a changed interval as changed.
Do it with the API
Every change through the API follows the same roles and is written to the audit log. $BASE is https://<your server>/api/v1 and H holds your token; see Quickstart: add a SQL Server instance with the API.
| Task | Request |
|---|---|
| See effective levels, and where each one comes from | GET /resources/{id}/checks |
| Set levels on one resource | PATCH /resources/{id}/checks/{check_id} with thresholds |
| Go back to the inherited levels | The same PATCH with "thresholds": null |
| Turn a check off or on | The same PATCH with "enabled": false or true |
| Change the interval | The same PATCH with "interval_seconds", from 10 to 604800 |
| See an entity's own defaults | GET /entities/{id}/thresholds |
| Set an entity default | PUT /entities/{id}/thresholds/{check_id} |
| Remove an entity default | DELETE /entities/{id}/thresholds/{check_id} |
The keys a level override can carry are warn, crit, expect, window and sigma. A key set to null turns that part off at that level.
# one resource: warn when page life expectancy stays under 900 s for 10 minutes
curl -s "${H[@]}" -X PATCH "$BASE/resources/$SQL01/checks/sqlserver.ple_seconds" \
-d '{"thresholds": {"warn": {"op": "<", "value": 900, "for": "10m"}}}' | jq .thresholds
# the Production entity: CPU warning above 75% for 15 minutes, no critical level
curl -s "${H[@]}" -X PUT "$BASE/entities/$PROD/thresholds/sqlserver.cpu_percent" \
-d '{"warn": {"op": ">", "value": 75, "for": "15m"}, "crit": null}'