What do you tell someone new to going oncall or PagerDutyl?

Question

[[ Based on tags and the wording this sounds like a PagerDuty-specific question, but it could apply to any tech or monitoring stack in my mind.... PagerDuty is just so dominant in SF-based jobs I've seen that is like tech Kleenex.... ]]

I've been searching in my spare time for a few months for an answer to: what do you tell a new recruit to pager duty (or PagerDuty or pagerduty)??? And I don't mainly mere "install the app and prey" training, but those are key parts of it. I'm more looking for generic, general, globalizable advice like:

• ask for help as early as possible
• don't go sleep deprived "too" many nights in a row ~3
• know who is likely to answer with low latency and who will notice work again at Monday 9am DOH!
• it is easier to find answer on StackExchange than random forums at 3am free plug
• triage: how big of a fire is it? how many fire trucks do you call in? Etc.
• when to merge incidents
• when to snooze incidents... but that is so scary!! "You don't get fired for snoozing here?" hahaha.
• good notification settings
• good phone/volume settings for getting woken up, even if you sleep like a log.
• scheduling and calendar layers
• rules of thumb and recommended tools
• know who to call for what subsystem (varies based on locality and org) but using a wiki or Confluence is often part of that

Maybe I'm searching wrong. I'm open to meta-suggestions for better ways to find this needle in the haystack. Searching for "starting pagerduty" makes sense to me, but it gets results of how to setup these things programmatically or technically, not how to instruct the humans in dealing with sorts of things. PagerDuty does try to address this, but it doesn't include the breadth of responsibilities that the job entails to me.

I also admit that I'm asking for everything and the kitchen sink and the city water supply for a bonus round. Getting little pieces of this as links and pointing folks at it would be a huge improvement over what I've found so far. I doubt anybody has done all of this in one doc or would ever want to do so.

I've actually already turned this question into an internal class. I'm happy to get permission from management to share my fun version of this after I pull out the proprietary bits. I'm still holding out for it having been done at some point before and I'm just too lazy to go that far into the search results. :)

Answer 1

There are a number of things that I think it's important to know before going on-call:

1. How to triage an issue. There should be a defined list of priorities with examples so that you can identify how serious a problem is.
2. What the priority levels mean. Can you ack this and then go back to sleep?
3. Who to contact post-triage. The on-caller's primary responsibility is triage, not fixing. Fixing the problem is good, too, but once you've triaged the issue, then it's up to you to rope in other folks from other teams and manage communications between them.

People jumping straight into fixing problems before triaging is the most common mistake I've seen (because, after all, we're used to fixing things and think of that as our job). But communications are often vital, and doing things in the wrong order can mean customers are left in the dark because you got caught up in trying to fix a problem before letting your support people know there is a problem.

Answer 2

Here's my two pieces of advice:

1. Work out schedules so that you're not on-call 24 hours a day. There should be a hand-off between daytime operations and after-hours support. I encourage people to split half their regular time between support weeks and project weeks; after-hours support should come during a project week.

2. Get in the habit of taking notes during an incident. That's the most important thing; even more important than "fixing" the issue. If bandaging the service requires you to stop taking notes, then it's time to call in additional support. There should always be someone taking notes.