How to mark a check as warning or error if a check fails multiple times within a certain time range using Sensu?

Question

A certain check works, but it should be checked multiple times within a certain range before a user gets notified.

There are multiple Sensu attributes that could be configured. This list has been consulted, but no solution was found. Is it possible to mark a check as warning or error if a check fails multiple times in a certain time range?

Perhaps aggregates would be an option, but should this not be used for multiple clients instead of using it for one check?

{
  "clients": 15,
  "checks": 2,
  "results": {
    "ok": 18,
    "warning": 0,
    "critical": 1,
    "unknown": 0,
    "total": 19,
    "stale": 0
  }
}

Answer 1

You want to use the occurrences filter, which is included by default in Sensu.

First, apply the filter to your handlers:

{
  "handlers": {
    "email": {
      "...": "...",
      "filters": ["occurrences"]
    }
  }
}

Then, in your check, you can use the occurrences attribute to only trigger the handler once the check fails or warns a certain number of times in a row:

{
  "checks": {
    "check-http": {
      "...": "...",
      "occurrences": 2
    }
  }
}

Answer 2

https://sensuapp.org/docs/0.28/reference/plugins.html#check-definition-attributes

The occurrences as defined in the other answer was required (+1). The refresh and interval were needed as well.

Refresh and occurrences had to be assigned to a check. It was not needed to change the handlers.

"a-check": {
  "command": "echo hello"
  "occurrences": 6,
  "refresh": 60,
  "interval": 10
},

https://gist.github.com/calebhailey/8a30a00c6aadfebf7f767444f0a3df49

The interval means after how many seconds a check is executed. The refresh means that after 60 seconds a notification is sent and the occurrences mean that if within 60 seconds the event was triggered 6 times that a notification will be sent.

It seems like the formula is as follows:

occurrences * interval = refresh

10 * 10 = 100