Is it possible to only have an alert triggered after a certain number of incidents from an integration?
For example, if my application reports a single non-critical failure of a kind, that's worth-while to troubleshoot, but probably not so bad as to wake up somebody in the middle of the night.
However, if the application is reporting the same failure over and over again, it's a symptom of a bigger problem that somebody should look at ASAP.
Any ideas?
The easiest way to build this would be to have some sort of metrics-based alerting system like Prometheus, datadog, etc. Those systems allow you to have a counter that gets incremented and you can see how many incidents happened in a given period of time on a pretty graph. Most metrics systems will tie nicely to PagerDuty for the alerting piece. Having any of these metrics systems functioning will also give you a historic baseline for your systems that is often incredibly handy in figuring our what went wrong and when.
The threshold should probably be number of incidents over a given period of time. If you just pick an arbitrary number, say 100, for this you will always eventually get to that limit. If you see 10 incidents in an hour that could be bad for you.
The hardest part of this would be that I don't think PagerDuty makes it easy to access metrics about the number of incidents. You can look at pretty graphs in their UI, but there's no easy way to feed that into your metrics system. You would need to add a bit of code to the other places you have creating incidents and have them increment this counter.
The way we handle this is to run our alerts through Sensu first. You can configure a Sensu check to require multiple failures to alert, and then also configure the check's notify to get sent to PagerDuty. This way, by the time PagerDuty hears about it (and pages you), it has already passed the "N failures in a row" criteria.
If you need to check not for number of failures in a row, but rather N failures across a time range, then that's a good use case for adding elasticsearch or graphite or similar into the mix. Whenever there's a failure, log an error or increment a metric, and then your check can simply look over the time range and see if the aggregate is above the limit.
