3

We have a small team, and we're just getting started with PagerDuty. We have a very noisy service that has a lot of testing goign on around it. At times, we've had to turn the service off to keep pagers from blowing up, but then forget to turn it back on.

Is there any way to get notified when a service has been disabled?

Stuart Ainsworth
  • 998
  • 6
  • 15

1 Answers1

2

3-4 years experience with PD here.

PagerDuty will not monitor itself in the way you want, it's made to push notifications, calls, alerts and etc. As mentioned in the comments you should use maintenance windows, disabling a service means you know what you are doing, that's not something I have ever done except for 1-2 testing purposes for the last 4 years. If you are forgetting to enable the service.. then there is another problem that is not technical related, you should not be forgetting that step! I would suggest you don't disable the service, use maintenance windows, if the service is noisy, why put it in PagerDuty, monitor it when it's fully down, or use some other rule to do this. What I would do:

  1. Evaluate what is a critical alert.
  2. Should you be notified via pager duty for every single thing?
  3. Probably set more dynamic alerts, that the alert will be triggered only in specific hours(work hours). Link to PD documentation
  4. Try not to disable the Service, use maintenance windows.
  5. Decide what kind of escalation policies do you need/want.

EDIT: Never tested this, but PagerDuty has an API that possibly could be used to get a status of a service, but something has to listen/run that API GET request and send it to another PagerDuty Service, that can be used to monitor what you wanted, kinda. API Service Documentation

EDIT 2: I tested the API, if you use that method, you can use something else to monitor for xx amount of time if the service is disabled, by getting this data out of the request: "status": "disabled". For example any Monitor that can send a GET request and get this data, can be set to "If status: disabled for 1 hour, send PagerDuty alert". And you can have a separate PagerDuty Service for example that listens for this monitoring.

enter image description here

Kristian Kanchev
  • 334
  • 1
  • 8