Automatically create incidents from PagerDuty alerts

Weekly Update

For many of you, sits on top of PagerDuty. You get paged. You check if the error is concerning. If so, you pop Slack open and declare an incident. Then you paste a link to the error in the incident’s channel. Rinse and repeat for every alert. If you suffer through some flavour of this cumbersome, repetitive flow, our latest release is for you!

To save you doing all that manual work over and over again, we can now create incidents straight from your PagerDuty alerts.

So how does it work? You start by choosing the type of PagerDuty incidents you care about, i.e. the services and urgencies for which we should auto-create incidents. Whenever we see a PagerDuty alert that matches that configuration, we’ll automatically create an incident.

Importantly, that incident will be created in a newly-minted triage status. This is to give you the time and space you need to investigate the alert. When you have a clearer idea of what’s going on, you’re ready to move the incident out of triage. We’ll have posted a message in the incident’s channel, with three possible outcomes and next steps:

  1. “A new incident” means that we’ll update the status from triage to investigating - from here on, you can keep managing your incident in this channel like any other incident.
  2. “Part of an existing incident” will let you point to another incident that this alert should be merged into. We’ll take care of archiving the channel for you, and making sure that incident isn’t reflected in your Insights.
  3. “Not an incident” will tell us to dismiss this incident. Again, we’ll take care of archiving the channel for you, and making sure that incident isn’t reflected in your Insights.

All the PagerDuty alerts linked to your incident will be ready for you in the “Attachments” tab on the Incident Homepage. To save you from more repetitive manual clean-up, when the incident is closed, we’ll also go and resolve these attached PagerDuty incidents for you!

And for the final flourish of magic: while you’re working in an auto-created incident channel, we’ll notify you about similar PagerDuty alerts before creating new incident channels. No noise.

Happy days!

What else we shipped

  • 💅 Added the latest update to the "Recap" slack modal
  • 💅 Split the concept of "creator" (eg. API key) from "reporter" (eg. Milly) in the API
  • 💅 Show more information about recently created incidents when creating a new incident
  • 🐛 Stopped listing "Incident Reporter" twice when assigning roles in workflows

Operational excellence starts here