Alert Aggregation, Thresholds and Alert API : HowTo

Alert Aggregation, Thresholds and Alert API : HowTo

This document takes an example of an rule and goes through

  • Life Cycle of an alert
    • Aggregations
    • Thresholds
    • Interpreting alert on the UI
  • Alert over the API
    • Various use cases of using alert API
    • Understating the alert JSON API

Life Cycle an alert

The alert life cycle starts with the user creating a rule on rules page of the application.

Below is an example of three alert rules , where the user wants an alert to be generated anytime any user escalates his or her privileges on the host (using sudo command).

  • The user wants a severity 1 alert for 10 such privilege escalations
  • The user wants a severity 2 alert for 5 such privilege escalations
  • The user wants a severity 3 alert for each of such privilege escalations

Below picture shows the summary of the three rules

Digging deep into the Rule

Lets take the rule of "5 in one hour" to understand different pieces of the rule .

The first portion of the rule is title, description, and filter to alert on.

  1. Title has text and replacements. Replacements should match up to aggregations selected for the alert.
  2. Alert filter is the filter to match for firing there alert. In this case, its sudo command with start type.
  3. Severity of the alert generated

The second portion of the rule has the options to select aggregations and alert thresholds

  1. The aggregation defines the uniqueness of the alert. In this case, since user, exe and arguments are selected as aggregations, if the same user uses the same exe with the same arguments a second time, it will be considered as the same alert. If any one of those aggregations change (ex - user uses same exe but different arguments, it will be considered as a different alert.
  2. The threshold counts the number of times the event matches with the filter defined along with aggregations, and generates an alert only after the count. In this case the threshold is selected as "5 events in 1 hour" it means if the same user uses the same exe with same arguments 5 times in an hour , then generate one alert.
  3. Time window : Please note that if the same event is generated the next time with in the time window, we do not generate a new alert, but we update the last alerted time of the same alert and attach that event to the alert record (see digging deep into the alert UI section to learn about the last alert time of the alert)

Alerts on the Alert UI

Here is how it will show up when the alert is triggered

Digging deep into the Alert UI

When the user clicks on the alert, the contributing event detail window appears on the screen. This will help the user investigate the issue.


Created Date and Last Alerted Date

When the user clicks on the alert,  the contributing event details are displayed at the bottom of the alert page.  The created time stamp (the timestamp of the first event contributing the alert) and the last alerted date (timestamp of the latest event contributing to the alert) are displayed on the details.  The user can choose to look at other contributing events (at most 5) of the alert as well by clicking "View Contributing Events" link.   When the user clicks the link, the last 5 contributing events are displayed in chronological order with first event on top.



Alerts over the API

How do I get alerts over the API ?

The following are various options

  1. Getting all alerts (on first page) : curl " -H "Authorization: <deploy_key>" | jq. Note that we only send alerts in pages, each page by default sends out 100 alerts per page.
  2. Getting all alerts on nth page : curl "" -H "Authorization:<deploy_key" | jq
  3. Getting alerts from specific start time to end time : curl "" -H "Authorization: <deploy_key>" | jq. Note that the start and end has to be in UTC and ISO 8601 formats. The following are examples of the format 2016-07-13T16:43:18+00:00, 2016-07-13T16:43:18Z, 20160713T164318Z.

Note that alerts , along with contributing events are available over the UI or API for a period of 1 year.  If an alert is dismissed, it is still available in the dismissed tab.

Anatomy of the Alert over the API

The alert API has 3 portions,

  • Meta data related to the alert
  • Events related to the event
  • Rule that generated the alert

The alert and events portion

The rule portion

Have more questions? Submit a request


Please sign in to leave a comment.