Vigilfield Docs
Detect and alert

Alerts

Turn a rule's matches into alerts, shape them from the query, and work them from open to resolved.

An alert is a match that someone should look at. Rule alerts come from a rule's rows. System alerts come from Vigilfield itself. Both appear on the Alerts page.

Turn alerting on for a rule

  1. Open the rule and click Edit, or start a new rule.
  2. Under Alerting, turn on Alert on matches.
  3. Pick a Severity: Informational, Low, Medium, High or Critical. Every alert the rule opens carries this severity. It is set per rule, not per row.
  4. Under Destinations, tick up to 5 alert destinations. At least one is required. If none fits, click New destination to create one in a new tab (see Alert destinations).
  5. Save the rule.

With alerting off, a rule's rows are only written to its result table.

Through the API, alerting is the rule's alerting object: a severity and up to five destination_ids, with no id empty or repeated. On an update, send "alerting": null to turn alerting off. Leaving the key out keeps alerting as it is. See Edit a rule.

How a run becomes alerts

Only scheduled runs alert. When a run of an alerting rule succeeds:

  • The first 10 rows each become one hit, in the order the results show them. Sort the query with sort or top to choose which 10.
  • A hit on a key that already has a live alert is added to that alert. Live means open or acknowledged. Its hit count goes up, and its last-seen time and latest row change. Nothing new is sent to destinations.
  • A hit on a new key opens a new alert, and the alert is sent to each of the rule's destinations.
  • More than 10 rows adds one hit to the rule's overflow alert, titled "More rows matched than were alerted: " followed by the rule name. Every row is still in the run's result table for seven days.

You shape each alert from the query with three reserved columns, alert_key, alert_title and alert_description. They are plain text built with strcat. There is no templating language. The rules for each column, the fallbacks when one is missing, and a full example are in A rule can alert. The short version:

SecurityEvents
| where Result == 'Fail'
| summarize failures = count() by Account
| where failures > 5
| extend alert_key = Account

Without alert_key, Vigilfield hashes the whole row, so a row whose count changed opens a different alert. Set a key when you want repeats to land on one alert.

What an alert shows

Open an alert from the Alerts page to see:

  • The title, severity, status, hits and last seen, at the top.
  • The origin: rule:<rule id> for a rule alert, or system:<kind> for a system alert.
  • first seen. First and last seen are when Vigilfield recorded the match, not times taken from the row.
  • Description, shown as plain text.
  • Latest matched row, as JSON. A row larger than 32 KiB is replaced by a note with its size.
  • Result table, for a rule alert: a link to the result table of the latest run that hit it. The link stops working after the table's seven days.

Title and description are cut to fit, never refused: a title at 256 characters and a description at 4096 bytes.

Status lifecycle

An alert moves one way only:

StatusMeaning
openNew. Hits on its key are added to it.
acknowledgedSomeone is on it. Hits on its key are still added to it.
resolvedDone. The next hit on the same key opens a new alert.

On the alert's page, click Acknowledge or Resolve. You can resolve an open alert directly. Repeating either action is harmless. You cannot acknowledge a resolved alert, and there is no way to reopen one.

Through the API, see Acknowledge an alert and Resolve an alert.

Find alerts

The Alerts page lists alerts newest first, by when each opened. Filter by status (Open, Acknowledged, Resolved), by severity, and by origin (Rules or System). Click Load more for the next page.

There is no total count. A page can hold fewer alerts than the page size, even none, while more remain.

Through the API, List alerts takes:

  • status: one or more of open, acknowledged, resolved, comma-separated.
  • severity: one severity.
  • origin: rule, system, or one origin such as rule:<rule id>.
  • limit: 50 by default, at most 100.
  • cursor: the previous page's next_cursor. Keep following it until it is null.

Who can see an alert

  • A rule alert belongs to the rule's team, as it is now. Members of that team can see it and act on it. A team granted read access to a rule's alerts can see them but not acknowledge or resolve them.
  • A system alert is for org admins. Only admins can see it and act on it.
  • The org owner can see and act on every alert.

An alert you cannot see is reported as not found. API keys and other machine credentials cannot use the alerts API at all.

System alerts

Vigilfield raises system alerts for events an admin should know about:

OriginWhen
system:rule_failedA rule's run failed. One alert per rule while it is live.
system:source_unhealthyA source failed repeatedly and was quarantined. Ingestion from it is paused until an admin resumes it.
system:usage_thresholdUsage crossed a warning threshold.
system:usage_capUsage reached the cap.
system:storage_key_unusableThe storage encryption key cannot be used.

When a system alert opens, Vigilfield emails the org's admins. It can also send system alerts to alert destinations. An org admin or owner sets that list through the API with Set system alert destinations, up to 32 destination ids. It is empty until someone sets it.

The bell

The bell in the header shows the system alerts you have not opened yet, among the 50 newest. Rule alerts never appear there. Clicking an alert in the bell opens it and takes it off your bell only. Opening an alert from the Alerts page does not.

Delivery failures

Each destination gets a limited number of attempts per alert. When they run out, the destination's id is added to the alert's delivery_failed list in the API. See Alert destinations.