📣 Refer a customer and get $200
📣Do you know someone who might enjoy incident.io?Refer a customer and get $200
Responding to incidents can be stressful. You've got 14 tabs open with some logs, a Sentry, a trace, some database queries, and a couple of GitHub PRs. It's hard to keep track.
Attachments is our first step to solving that problem.
Attachments allows you to connect something from another system into incident.io. We're not just talking about putting a link on the timeline though, we think you deserve better than that.
Currently, you can attach PagerDuty Incidents and GitHub Pull Requests to an incident. But don't worry, there's more to come.
To attach something, you simply need to paste a link into an incident channel:
You can view attachments in Slack (via /incident attachments
) and in the Web UI:
Once it's attached, we'll help you out wherever we can.
We'll comment on the PR (if it's in a private repo) with a connection back to the incident, so it's easy to find in the future
When a Pull Request gets closed or merged, we'll let you know in the channel
To start using the feature, paste a GitHub link into an incident channel and we'll do the rest!
For many of you, incident.io sits on top of PagerDuty. You get paged. You check if the error is concerning. If so, you pop Slack open and declare an incident. Then you paste a link to the error in the incident’s channel. Rinse and repeat for every alert. If you suffer through some flavour of this cumbersome, repetitive flow, our latest release is for you!
To save you doing all that manual work over and over again, we can now create incident.io incidents straight from your PagerDuty alerts.
So how does it work? You start by choosing the type of PagerDuty incidents you care about, i.e. the services and urgencies for which we should auto-create incidents. Whenever we see a PagerDuty alert that matches that configuration, we’ll automatically create an incident.
Importantly, that incident will be created in a newly-minted triage
status. This is to give you the time and space you need to investigate the alert. When you have a clearer idea of what’s going on, you’re ready to move the incident out of triage
. We’ll have posted a message in the incident’s channel, with three possible outcomes and next steps:
triage
to investigating
- from here on, you can keep managing your incident in this channel like any other incident.io incident.All the PagerDuty alerts linked to your incident will be ready for you in the “Attachments” tab on the Incident Homepage. To save you from more repetitive manual clean-up, when the incident is closed, we’ll also go and resolve these attached PagerDuty incidents for you!
And for the final flourish of magic: while you’re working in an auto-created incident channel, we’ll notify you about similar PagerDuty alerts before creating new incident channels. No noise.
Happy days!
Today we're releasing our API to the world. It's been in early-access for a few weeks and it's been amazing to see what people have built with it, and we can't wait to see what other integrations you'll create.
You can use our API to declare incidents from other tools, export all your data, and configure custom fields, severities, and roles automatically.
See the full details on our blog, or dive into the API reference. We'd love to hear what you're building in the #api channel in our Slack community!
When it comes to private incidents, our top priority is keeping private incident data safe. Being able to harness the power of our whole product during a private incident is a close second.
It's now possible to build workflows that act on private incidents. We'll only let account owners make changes to these workflows, to make sure that other users can't use workflows as a way to gain access to sensitive information.
Along with that, we've also added a condition so you can choose to target only private incidents with your workflow (if you'd like).
You can learn more in our help centre.
We know you love exporting follow-ups into your favourite issue tracker. So much so, that a number of you recognised that we don’t don’t sync changes quite fast enough!
We’ve doubled down on our Jira Cloud integration, meaning we now listen for issue updates and sync these to follow-ups immediately. So next time you update the Jira issue update status to “Done”, this will also get ticked off in incident.io 💪
Time zones are confusing enough as it is - I should know, I was very confused when the clocks went forward on Sunday - so we weren't helping things by displaying timestamps on the dashboard in UTC for those of you in other parts of the world. Now if you hover over a timestamp (or tab to the timestamp with the keyboard), it'll display the timestamp in both UTC and your local time, in a format a bit more familiar to you (like MM/DD/YY for you folks in the US).
/inc actions
)@here
and @channel
in your Workflows in the Slack send message steps (thanks Yoco)Look, sometimes you can't get to solving all your follow-up actions right away, we understand. So maybe your follow-ups page was looking a little full? It's ok, we've got you, we've added the ability to filter the follow-ups page the same way you can filter the incidents page.
We also added a workflow step to invite a user from a PagerDuty escalation policy, and assign them an incident role!
Actually 17 more things...
Custom fields are a key part of incident.io, helping you catalog past incidents, analyse trends and integrate with workflows and announcement rules.
Today we're releasing support for three new types of custom field:
Supporting freeform text values, create fields that record the impacted customer, or perhaps any third-party services involved in the incident.
Especially for regulated industries, you often want to track the number of customers affected by an incident, or the amount of money involved.
Numeric fields support integer and fractional numbers, which you could use to create:
Imagine your company encourages creating a separate public post-mortem/root cause analysis, and you'd like to track that doc alongside the incident.
Or perhaps you create GDrive/Dropbox spaces for each incident, containing files used during the response.
You can now create a link custom field which is synced with a Slack channel bookmark on the incident channel, helping responders find the links during the incident and whenever anyone visits an old channel, too.
Custom fields is just one big change that went out this week, while another is our totally redesigned website!
Thanks to @jamesm for his amazing design work.
And beyond that, here's the full list:
We think incident response should be open for everyone, but there are some things that are best kept within a smaller group.
We've introduced roles within incident.io, so some organisation-level settings can only be changed by selected Administrators.
Users can have one of four roles: Viewers, Responders, Admins and Owners.
Viewers can:
Viewers will automatically become responders when they get involved in an incident.
Responders can:
Admins can:
Owners can:
If you'd like to take a look at your Users, check out our shiny new Settings page.
n.b. Until March 1st, this feature is opt-in for existing customers.
Here's what else we've been up to this week:
❤️ Happy Valentine's Day from all of us at incident.io ❤️
It's great to be able to track incident follow-ups alongside other work in your issue tracker of choice.
As a festive treat, we're announcing a new partnership for us at incident HQ. We've supported Jira Cloud for a while, but today we introduce support for Jira Server!
As with all our issue tracker integrations, once connected, you can export follow-ups to Jira Server from the web dashboard. We'll then keep the incident.io follow-up 'in sync' with your Jira board, so updates in Jira will be reflected inside incident.io on the follow-ups page.
You'll also be able to view more details on the insights page, to understand how your team is managing follow-ups.
Here's what else we've been up to this week:
Finding patterns in why things go wrong can be hugely powerful in working out where to focus your limited resources. It can also be a huge pain to dig into this data.
Today we're releasing a revamped Insights section that gives you far more access to dive into your incident response data.
We're starting off with three sections:
This is where you'll find high-level metrics like median time-to-fix, and a chart of incidents reported by severity:
As the old saying goes, "fool me once, shame on you, fool me twice, shame on me." Taking the steps to learn from every incident and prevent the same issue coming back again is crucial. Here, you can see the completion rate of follow-up actions over time, and break them down by severity. You can also see how many incidents are having a post-mortem written.
Everyone loves a little healthy competition. This tab shows you top incident responders: people who report the most incidents, lead the response, or just come to help out.
Here's what else we've been up to this week:
We strongly believe that incidents should be public by default, and that transparency is an important foundation of any good incident process.
However, we recognise that there's clearly exceptions where public incidents aren't appropriate, such as:
We've had a lot of requests from our customers to better support this use case, so we've built Private Incidents!
When creating an incident after enabling Private Incidents, a user can choose to make it private.
If an incident is private, we will:
We will not:
#incidents
channel, or any other channelsAny member of a Private Incident can invite someone else, either by inviting them in Slack or via the web dashboard.
You can also revoke someone's access if you need to: everybody makes mistakes!
Head over to your Settings and enable Private Incidents.
Now, whenever you declare an incident, you'll be asked whether to keep it public, or make it private.
As we've had so many requests for this feature, we're very excited to be releasing it! Take it for a spin, and let us know your feedback by joining our Community and sharing in the #feedback channel.
Here's what else we've been up to:
It's the new year, and we're all ready for a change. Some of us have been tidying up and clearing out the cupboards - our CEO Stephen got rid of 6 bags of clutter over the weekend.
The same goes for the incident.io app: it's easy to forget to archive incident channels and they can sit around for months after the incident is over. So we're happy to introduce auto-archiving as our newest feature! We'll automatically archive incident Slack channels for you after either 7, 14 or 28 days.
You can enable auto-archiving in the Settings section of your incident.io dashboard, in the "Automation" section.
Note: We can keep your Slack workspace tidy for you, but make no promises about your physical workspace.
Here's what else we've been up to this week:
Happy New Year, from all of us at incident.io!
Winding down for Christmas is a great time to tackle feel-good product feature requests.
This post will focus on the changes we made to workflows, allowing you to:
Let's dive in!
You can now temporarily disable specific workflows, making it easy to build and test a workflow without worrying it might run against on-going incidents.
To disable a workflow, press the "Disable" button at the top of the workflow page, then use the "Test" button to run against an incident of your choice.
When you're done, use "Enable" to reactivate the workflow.
You can use any incident for testing- we recommend creating a test incident
(/inc test
) exclusively for this purpose.
Lots of companies have mailing lists that should be alerted when serious incidents occur (execs@, red-alert@) and for subsequent updates.
You can now build workflows to send these emails, using the editor to template incident data into the email.
Start with the Urgent incident template, or create your own with a send email step like:
Here's an email sent with this configuration:
Use this to subscribe executives or senior leadership to incident updates, or have teams hear about incidents of a specific type (ie, the Legal team for all Data breaches) by filtering on incident Custom Fields.
People want to hear about urgent incidents immediately, not when they next check their email inbox.
For this, you can now create workflows that send an SMS message, including details about the incident or instructions.
Start with the Send SMS message template, or create your own with a send SMS step like:
Here's an SMS sent with this configuration:
If you're looking for a lightweight pager, this should do the trick! And if you want access to a workflow step that phones people, please let us know and we can enable it for you.
Last but not least, you can now use workflows to automate assigning incident roles.
Depending on your company, you might want to:
Create a workflow with an "Assign incident roles" step, like this:
This would assign the person that reported the incident as the Incident Lead — this ensures you've always got someone leading the incident.
That's it for workflow related changes we released over the break.
And as usual, here's a list of the smaller ticket items:
From today, we're releasing an interactive tutorial that can teach people the basics of using incident.io, including:
/inc update
)/inc lead
)/inc actions
):pin:
)/inc close
)You can start the tutorial by typing /inc tutorial
in any of your Slack
channels:
Completing the tutorial takes about 5 minutes, quick enough to do just before joining an incident, or during a coffee break. Becoming more familiar with incident.io means you improve how you respond to real incidents- definitely worth a few minutes to try!
Just as responders can benefit from familiarity with the Slack app, it's important to understand the incident homepage, and how actions in the incident channel are reflected to anyone keeping track via the website.
For that reason, the tutorial ends by encouraging people to visit their incident page, where we'll trigger an interactive tour that explains each section:
Next time someone asks "how do I use incident.io?", give them /inc tutorial
for a personalised walkthrough.
This is our last official changelog of 2021, but stay tuned for some feel-good content to enjoy during your Christmas break!
/inc tutorial
/inc update
flow, auto-fill the incident summary from the update
if not set, and provide the last update for reference when writing the
summary.Now you can customise your workflows even further! We've supercharged workflow message steps with a rich text editor, allowing you to format your messages easily, add links, and - most excitingly - add special variables into your messages.
Choose from a whole host of context-dependent variables, including information about the current incident, message and/or user (depending on the trigger you’ve selected).
You can also use Slack's mrkdwn syntax to take advantage of all the formatting options that Slack provides.
If there are any variables we haven't included that you'd like to see, let us know over at the Community!
After many weeks of work, we're delighted to announce the latest feature of the incident.io platform: Workflows. Configure your processes once, and we'll make sure you follow them, every time ✨.
Workflows allow you to automate certain actions and behaviours based on specific triggers. We provide a set of building blocks which you can put together to encode your organisation's processes into incident.io.
That means you can now:
Check out our blog post for more details on what Workflows can do, or jump straight to the app to get started.
If you use Zoom, check out our docs on how to get setup in your incident.io workspace.
Our list of integrations grows larger this week with the addition of Zoom, the video conferencing product used by a number of our customers.
Installing Zoom will enable the "Automatically create incident call" settings in the dashboard. If configured, incident.io will create a Zoom meeting whenever an incident is declared, which reponders can use for high-bandwidth discussion.
Creating a call link has always been something we advocate for at incident.io, especially in an increasingly remote workplace. We're excited to get this released, and look forward to increasing the depth of the integration now our app is officially part of the Zoom Marketplace.
We're awaiting review for our application to the Google Marketplace (for Google Meet), and expect to see first-class support for video-conferencing in our upcoming workflow product.
Most of our focus was on building the first version of workflows, which we managed to get to a working prototype! 🎉
But life can't always be about workflows. Here's a few of the changes we snacked on alongside our main:
You can now close incidents from the dashboard (instead of having to be in the relevant incident channel). This is particularly useful when tidying up old incidents ready for analysis. It also continues to bridge the gap between what's possible from inside Slack vs. the dashboard.
When we're working on a big feature like announcement rules (and the workflows engine which powers it), we work hard to stay focussed on the task at hand. This can make us feel uncomfortable as we constantly come across things that we'd like to improve: either from customer feedback or from our own usage of the product.
To make sure we stay focussed on the big ticket items, but not letting our backlog of polish and bug fixes grow uncontrollably, we schedule time between big projects to knock items off the list. We call these wiggle weeks (an idea inspired by a great Intercom blog post). As you've probably guessed, this was one of those weeks.
Starting today, you can use the incident.io dashboard to create Announcement Rules, allowing you to subscribe a collection of Slack channels to announcements for incidents matching specific conditions.
incident.io has always shipped with a default #incidents announcement channel, which receives a post whenever an incident is created.
While our users often say how much they love this channel, with the increase in visibility it brings, not everyone wants to hear about every incident.
Perhaps your exec team only want to hear about incidents where the severity is greater than Major?
Or maybe Compliance need notifying whenever an incident of type Data Breach has been created?
That's where Announcement Rules come in.
Up until now, we'd announce each one of your created incidents in one channel - #incidents. Now, you can choose additional channels to post certain types of incident into - and define exactly what those criteria are.
From the incident.io dashboard, create rules that match against incident dimensions such as:
Affected Teams
includes DataWhenever incidents are created that match these conditions, we'll send an announcement post to the channels configured on the rule.
Once the rule is created, we'll notify the channels involved to let them know they've been subscribed.
And as a final sweetener, an often requested feature is the ability to change the default #incidents announcement channel.
This behaviour is now controlled by the default announcement rule, which - just like other rules - you can configure from the settings dashboard:
That's all folks! It's worth noting that announcement rules are the first of a few features powered by our underlying workflows engine, which we're hoping to expose as a first class product feature over the next month.
Watch this space!
And in the typical weekly round-up:
We've been working hard to stay focussed so the team has been deep in announcements, but we've picked up a few other bits along the way:
This week we've made it even easier to filter your incidents on the web! Maybe you want to view only the incidents assigned to you as incident lead, or show any incidents with a severity of Major or greater. Or BOTH. So many filtering possibilities!
Where previously you could choose to show incidents from the last 7 days, 30 days or 90 days, we've added some more complex date filtering as well, including showing all incidents within a specific date range.
You can even copy or bookmark the URL of the filtered view to share it with others.
💅 We introduced some new and powerful incident filters, improved date filtering, and added filtering by assigned roles.
💅 You can now disconnect integrations from the Settings area!
💅 Incident postmortem docs now have a link to the incident's Slack channel
💅 We now show a warning when you're updating your public status page from within Slack, to make sure it's what you absolutely want to do!
💅 We've now got a shiny new Pricing page!
🐛 Changing custom fields on the incident detail page will now update the UI afterwards.
🐛 We fixed a bug where the Auto Close Incidents setting wasn't loading any content.
👷 Sophie joined the team as our newest engineer! She'll introduce herself soon - for now she's been picking up some of the things you've been reporting in the incident.io Community.
This week we bring you a brand new way to declare incidents from the dashboard, as well as a whole host of improvements, polish and bugfixes!
We also recently announced our new $4.7m funding round, led by Index Ventures and Point 9, and moved to a new office to support our rapidly growing team. It's been a busy few weeks, and we're currently shipping product surrounded by cardboard boxes and cables we can't quite place, but we're not slowing down!
It's the end of an era today, as we're moving out of @incident_io HQ v1.0 👋 pic.twitter.com/p6CZcPRlAf
— Chris Evans (@evnsio) October 6, 2021
We've also registered a neat little domain - inc.new
to make it even easier to do it, just type inc.new into your browser, and it'll take you straight to the "declare an incident" screen - give it a try!
As always, we're excited to hear what you think about what we're shipping, and we rely on your feedback to shape what we build next - let us know your thoughts in the incident.io community Slack!
This week we're launching an early preview of a feature we're calling Decision Flows, which aims to make routine decision making easier and more transparent during incidents.
It all started two weeks ago, when a late night YouTube binge led me to videos about GDPR (welcome to my life!) and it reminded me of an interaction I had with the DPO at a previous company. They wanted engineers to assess whether an incident might need their involvement, and to help them decide they shared a document with a list of criteria. It seemed reasonable, and in isolation it might be doable, but we had multiple similar requests from different corners of the company: marketing asking to be looped in when the customer impact ticked certain boxes, Risk and Compliance sharing criteria for regulatory reporting, specific guidance for when to put up the public status page – the list goes on.
Soon, you have folks needing to consult 13 different documents to make the right decisions, and they're so focused on doing the right thing they forget they have an incident to resolve.
This got me thinking: what if instead of jumping out of Slack to follow a doc, you could be asked a sequence of questions to help make the right decision? A few hours later, I had a scrappy prototype built in Slack to see how it felt...
I've been working on a new @incident_io feature to help with complex decision making and guidance in incidents.
— Chris Evans (@evnsio) September 17, 2021
e.g. Rather than check a doc to know if you should report a GDPR data breach, your Data Privacy team can configure a Decision Flow for you to follow, right in Slack. pic.twitter.com/ZJMeRc5l3N
We're excited to see what you do with them - let us know in the incident.io community Slack!
There's work in the pipeline that we're excited to share soon, but for now here's some of the other things we've shipped:
This week we're doubling down on Jira; helping you connect incident.io to your existing processes.
We know that a lot of our users use Jira, for a wide variety of use-cases surrounding incidents. We want to make it really easy to integrate incident.io with your existing processes, so that its simple and safe to start using incident.io. It also means you can keep using all your existing reporting processes without having to start all over again.
You can now configure your incidents to create a ticket in Jira whenever an incident reaches a pre-defined severity, by editing your automation settings.
The ticket will have a link back to the incident.io homepage so you can easily move between the two:
P.S. We want to make sure you don't get an avalanche of Jira tickets, so test incidents will never create Jira tickets.
We've spent a lot of the week working on our internal debugging tooling, so we can identify and fix issues quickly. We've also worked on a few things you might notice:
💅 We now support freetext fields when exporting to Jira
💅 The Jira export modal now walks you through selecting a project and then an issue type, so it's easy to see what fields you need to fill in.
🐛 If you pin a message that's in a thread, it'll now show up in the timeline in the dashboard and Postmortem
🐛 If you pin a message from google hangouts, that will now get added to the timeline
👷 We made our error reporting, stack traces and logs much more useful internally, to help us fix any issues that crop up even faster!
👷 We invested some time into optimising our build and release pipeline, we can now ship changes to you in half the time 🚀
With last week being just four days, we decided to tackle some small quality-of-life changes we know people have been dying to see.
After our recent introduction of follow-up actions as a mechanism for tracking post-incident tasks, we've seen an uptick in people exporting their actions into external issue trackers.
If you're a Jira power-user, you may have struggled to export your actions if the project or issue type you were exporting into required custom fields.
After digging into Jira's API and getting to grips with the various different field types and auto-completes, we now support custom fields of whatever form you may have configured, all inside the incident.io dashboard:
Exporting actions into backlog issues is something we do ourselves at incident.io, pushing our post-incident tasks into Linear. We keep track of the link between your incident action and the external issue, and sync action states/assignees so incident.io reflects the current state of the issue.
Doing this allows us to provide you insight on how well you're actioning your incident follow-ups. View a summary of outstanding follow-ups in your dashboard or export to a CSV if you want to crunch the numbers yourself.
We'll be looking to do more with this data in future. Watch this space!
Loads of our users want to hook into incident creation to and customise the messaging given to responders, perhaps offering guidance or advice that might be useful to help get started.
As of this week, you can configure your incidents to:
You can control the message and invites in your settings page:
That config would create incidents like this:
We want incident.io to be fully adaptable for your organisation's processes, as everyone has different wants and needs when it comes to running incidents.
These two features are a preview of a more powerful and general concept we want to build into the product, something we're calling workflows. We expect to roll these settings into workflows once it's ready, so watch this space!
If you want to follow this work or see what we plan on hitting next, we've updated our product roadmap with our most recent plans.
Please send us any feedback or suggestions of what you'd like to see us build in the Slack community- we always want to hear from our users!
As always, a quick summary of what we released:
💅 Rejig order of incident creation messages for improved readability
💅 Speed up our incidents endpoint, helping the dashboard load faster
💅 Updated the public roadmap with our most recent plans
👷🏽 Building out interviews for a variety of roles- checkout our jobs page, we're hiring!
🐛 Support post-mortem copy-to-clipboard in Safari
🐛 Clear the actions edit modal of unsaved changes
This week, our focus is all about getting data out of our product to help out with analysis, reporting and tracking. You can now export data to a CSV from the incidents and follow-ups page.
From the incidents page you can export a CSV of the incidents that you're currently looking at. It will include all the information you see on that page (status, severity, name, description) as well as custom fields and roles you have configured.
From the follow ups page you can export a CSV of all the follow up actions across your organisation. This will include information about the action (description, reporter, assignee) and the external issue tracker reference if you're using one. There's also information about the incident on each row to help you sort, filter, report and apply your spreadsheet wizardry!
/incident fields
as well as /incident field
to update custom fields.incident.io provides actions, our lightweight toolkit for tracking and assigning work during incidents.
After speaking with our community and collecting feedback, we spent the last week making a number of improvements to actions that we hope will better fit your incident workflows.
We believe there are two types of action you create during an incident – those that need doing now, and those that should be followed-up after an incident has been closed.
Actions that need doing now might be:
While follow-ups could be:
You can now mark actions as follow-ups, helping separate work that needs doing now from tasks that can be handled post-incident.
Create follow-ups in Slack like any other action (/inc action
Thing
) but with "When does it need to be done?" set to "After the
incident":
Sometimes you'll create an action, then realise it's best handled as a
follow-up. Moving actions into follow-ups (and out!) can be done from the /inc
actions
modal:
Once created, follow-ups can be managed in your incident dashboard, where they are shown separately from your other incident actions:
We expect people will review their actions as part of closing an incident, taking the opportunity to promote an action to a follow-up if it's still relevant.
If this is how you use them, we suggest exporting any follow-ups to your issue tracker (see the "Export to Linear" button) to link against your ticket, ensuring incident.io will sync the follow-up to your issue tracker.
Following this workflow will mean you can use the new "Actions" view to track outstanding follow-ups across all your incidents, giving you a picture of how your organisation is doing on their post-incident work:
While follow-ups are this weeks big news, we took the opportunity to improve actions more generally, too.
Expect some major quality-of-life improvements when working with actions from within Slack, such as:
/inc actions
defaults to showing outstanding actions only,
assuming responders will want to clearly see what remains (rather than what
they've already done!)See it in action here:
Lots! Mostly covered above, but a few small tweaks
🆕 Follow-up actions to track work that happens after closing an incident
🆕 New actions tab for viewing outstanding follow-ups across all your incidents
🆕 Mark actions as not-doing, helping responders focus on what they need to do now
💅 Simplify the /inc actions
modal and clearly separate actions by status (Slack)
💅 Provide an action edit modal for changing action details (Slack)
💅 Show quick-access buttons to transition actions between states (Slack)
💅 Separate follow-ups from in-incident actions (dashboard)
💅 Use consistent action status names (Open, Outstanding -> Open, etc)
💅 Filter incidents by "has outstanding actions" or "has post-mortem" (dashboard)
💅 Now display severity descriptions in the incident create modal (used to just be in the update modal, now its in both) - thanks Tiago!
💅 Bookmark your favourite incident filters, so you can share your search queries
💅 Custom fields now appear in the incident recap view
👷🏽 Continue porting API endpoints to goa, allowing us to type-check our frontend code against our backend implementation
🐛 Clear create action modal inputs after successful create (dashboard)
🐛 Notify incident channel whenever actions change in the dashboard
🐛 Fixed attribution of timeline items when you pin another persons message
We've added filtering so you can more easily find the incidents you're interested in.
On the incidents page, we've still got the free text search, but have added some more filtering options. This enables you to look at incidents with particular characteristics; either to find a particular incident or to help do some high level analysis. We also think this might be useful for running regular incident reviews and for anyone wanting an 'at-a-glance' view of the activity over the last week or month.
Currently you can filter by:
There'll be more filtering options coming soon - please let us know what you'd like to see next!
P.S. I really wanted to call this blog post ge-filter fish, but we decided that too few people would understand the reference. This post-script was the compromise.
incident.io ships with a set of commonly used fields for an incident, such as the severity and who's leading. However, a feature that virtually all of our customers have been asking for is the ability for them to add their own custom fields to incidents. They want to be able to track things that are specific to their business.
We're really excited to release Custom Fields to do exactly this. Custom Fields are a way for you to define the data that you want to collect on your incidents, and have incident.io collect it for you before, during or after an incident.
Here's a few examples of useful Custom Fields that we commonly see:
For defining what data you collect, we currently support:
We'll shortly support collecting text values, number, booleans and dates.
We're really excited to have shipped this. It's another step towards incident.io becoming the single source of truth for all of your incident data within your organisation. We can't wait to see what you build with it.
To get started, head over to your Settings page and add your first Custom Field!
On top of the usual collection product improvements, this week we've also shipped our new blog. The old one felt a little uninspiring, and since we've got lots we want to share, we thought a little overhaul was a good investment in time. To mark the occasion we shared the first in our interview series, this time featuring Colm Doyle from Slack.
We've had a few things competing for our time over the last few weeks, which has slowed the pace of change a little. Next week's looking pretty clear though, with plenty of time to devote to product and engineering. Stay tuned!
/incident update
, we now prompt you to check that the overall summary still makes sense. A well written summary can go a long way to keep folks on the same page.Over the last week we've been busy making it even easier for your whole organisation to onboard with incident.io. Whether it's simple step-by-step instructions for the installer, or helpful messages to folks after they've participated in their first incident, we've got you covered.
/inc update
, we'll now prompt you to check the incident summary still makes sense. A few minutes getting everyone on the same page can save hours later on./inc update
command whilst also changing the severity, we'd show the old severity in the closure modal. Now, thanks to a report from the folks at Ziglu, we show the correct one.Instead of doing just one thing last week, we made lots of smaller improvements to the product. We also got a lot of internal admin off our plate to keep the paperwork gods happy and keep the wheels turning.
We won't bore you with the paperwork, but we hope you enjoy the new features and improvements!
/incident status
will still work just fine.Incidents are a team sport, and to help everyone get involved we built incident actions. With a quick /inc action
in Slack, anyone can view, create, and pick up actions, making it easy for folks to self organise around whatever needs to be done to get things back on track.
We got some feedback that the actions interface in Slack could be a little clearer though, for example the layout was a little cluttered and it wasn't easy to see who created what. We've now refined the design and tweaked what we display so it's even easier to get a clear picture of what's going on.
Got thoughts on this or any other part of incident.io? Jump into our community Slack and let us know!
/incident help
from outside an incident channel, we used to open the create incident modal with an incident title of "help". You were probably after some actual help, so we now show you modal with some tips. Thanks 100/x for this feedback!We're fanatical about great support here about incident.io. Every customer gets a shared Slack channel to make it easy to ask questions and give feedback. We've now made that even easier by adding Intercom to incident.io.
Whether it's a question about how to add a custom incident role, or feedback on an improvement we could make, you can now send that to us in a couple of clicks. Click the button in the bottom right hand corner of incident.io to get started.
Behind the curtain, Chris, Pete and Stephen will be diligently answering your questions, fielding your feedback and blushing at your compliments. Looking forward to hearing from you!
Good incident response is about good communication. In incidents, it's particularly helpful to provide a small, structured update to the organisation on a regular basis. This keeps everyone on the same page, and helps resolve worries about when the next update will be coming, too.
We're always looking for ways to make good communication easier, and a few customers have mentioned that using seperate commands to udpate the severity, status and summary often felt a bit harder than it needed to be. You might not remember to update the status, or forget to update the summary after new information came to light — we certainly did!
We think all these fields work better together, so last week we set to work on incident updates!.
Updates are an easy way to provide a small, structured update to the rest of the organisation on a regular basis. We tie together existing fields like Status and Severity, with a new field allowing you to share any other context. Additionally, if you set a reminder for when you're expecting to send the next one, we'll remind you at just the right time ⏰
Hopefully this makes internal communications a lot easier, and reduces some of the burden when it comes to communicating regularly. Give updates a spin in your next incident, and let us know what you think!
It's a been a while since we last posted one of these, and despite the radio silence we've been busy behind the scenes at incident.io HQ. Firstly, we now actually have a HQ, and it's in an old firestation - very fitting. Pete and Stephen have left their full-time jobs and are now working 100% on incident.io.
This week we've decamped to an eco-lodge in the Wye Valley to plan for the next few months. We've got hundreds of ideas and a mountain of valuable feedback, so we're taking a week to refine it all into something amazing.
No more breaks now. Expect weekly changelogs to be weekly. All systems go.
We received some lovely feedback from customers over the past week — this kind of thing makes our day, thanks folks!
We at @Farewill started working with the fantastic @incident_io team earlier this year, and honestly it makes life SO much easier to have seamless tooling when everything is on fire. Watching our engineers working in the incident channel to check how we're affected is just 🤩
— Sally Lait (@sallylait) June 8, 2021
Check out @incident_io to manage incidents within slack. Awesome workflow and all the automation you ever wanted to bring more structure and fun into a potentially stressful situation. We'll start using it for every alert @giantswarm now.
— Timo Derstappen (@teemow) June 12, 2021
Super awesome demo of @incident_io from @sjwhitworth earlier. Really excited about what the team is doing to switch up incident management by bringing together tools we use & love into a streamlined workflow🔥🔥
— Nick Coates (@nickcoatesuk) June 10, 2021
We now have a number of customers using us daily to respond to incidents, and we're seeing more of you leaning into improving your post-incident followup. This is an area where until now we've been heavily opinionated, without much room for customisation - something we always knew we'd need to change.
We want to make it easy to imprint your process directly into incident.io - we want you and your team to feel like this is your tool, not someone else's.
We've always been able to automatically generate incident documents (also often referred to as post-mortems) and timelines automatically, but now you can now customise the format to your heart's content.
closed_at
timestamp that we use for tracking. That was silly. It now doesn't do that. Thanks for helping us spot this one, Farewill!In incident.io, roles are a first class citizen. We think of them as 'hats that you can wear' during an incident. We use them to make it clear:
Until now, we only supported a single role - the Incident Lead. We knew this was limiting, and our customers agreed, so we fixed it.
You can now add and customise your own roles to fit how you like to respond to incidents, attaching a description and instructions on how to play the role. Roles can be assigned and re-assigned during incidents by using /incident roles
.
When someone is assigned to a role, we'll privately send instructions to the assignee to make it clear what they need to do, and help them get up to speed.
Head over to the Help Centre to learn how to use them, or join us in Slack to let us know what you think!
/zoom
in an incident channel, we'll prompt you to add it as the call for the incident. Like magic.@incident
in our codebase.When we started incident.io, we felt passionately that we should replicate the feel of human communication during an incident. Need to escalate to someone? Just talk to us with @incident escalate
and we'll guide you through it. The drawback? We made a lot of noise in your incident channel 🙉
We've listened to your feedback, and today we're making the move to Slack's slash commands. Whilst it's just a single character, moving from '@' to '/' has a big impact. When you /incident escalate
it makes no noise in a channel, and as an added bonus we can take you straight to the right place. It's a win-win!
As if that's not enough, we're also introducing the Incident Home. Can't remember the command for the status page? Just type /incident
(or /inc
if you're in a hurry) and we'll help you get to the right place.
Most of the energy around incidents is, quite rightly, focussed on coordination and solving the problem, but there's a tonne of value in what happens afterwards, too.
Whether it's to spot topics for a postmortem, or learn which parts of your incident response could be improved, just asking "how did we do" is a powerful tool.
Often we forget to ask, and it involves a lot of additional admin to follow up and collect responses, but from today, incident.io will handle it for you.
We'll automatically check in at the end of an incident and ask everyone how it went, with a few specific questions, as well as some more general prompts.
This is just a first pass, but we've already seen customers using it to gather ideas for improving their incident response. Let us know what you think!
Our main focus this week has been making sure it's easy to learn what incident.io is, and how it works. We've spent a lot of time working on great tooling for incident leads. However, by definition, there's only a single incident lead but many participants and observers. We want to make sure you have a great experience too!
As part of that, we've polished the messages that you get when joining an incident channel to make them clearer and more concise. Additionally, you can indicate whether you're there to help (we'll announce it in the channel if you are), there to observe, or if you want a quick introduction to incident.io.
We'll spend the rest of this week focusing on improvements to onboarding: look out for some new features coming out over the next few days!
lead
, rename
, doc
and call
. Give it a try with /incident lead me
. Only a few left to go now../incident action
, we'd send two separate updates into the channel. We've combined that into one update. Thanks for the feedback, Upvest!/incident summary Hello world
, we used to discard Hello world
. That now forms part of the update that appears in the modal. Sorry about that.Our customers want a way to train people on incident.io, without unnecessarily worrying others in the organisation that things are going sideways or messing up their lovely incident library and statistics. Until now, you had to use normal incidents to do this. That's no good.
Well, enter test incidents! Test incidents are exactly the same as normal incidents, except:
We think that they're best used for:
Kick off an incident by /incident test
and let us know what you think!
First impressions are important. We've seen more and more people using incident.io recently, which has been fun to see, so we've used this influx of new faces as a prompt to think about what we could do better here.
We're currently focussing on onboarding and adoption to make it easier for new users of incident.io to get up to speed, whether that's when you install us, or the first time you're pulled into an incident.
We'll have more to show next week, but we've already made a start, so you'll probably notice a few changes. If you spot them, let us know what you think!
/incident
to kick off an incident? Too many characters and you're falling asleep by the end? Well, you can now type /inc
to save you time, and effort.close
, recap
, statuspage
and escalate
. Try /incident recap
for a quick spin!/incident The website is down
, we used to ignore The website is down
and make you type it again. We now automatically populate that into the modal, to save energy for your precious little fingers. Thanks Ravelin, Bud and many others for this feedback!In 2021, most companies have a status page set up to make it easier to publicly communicate the real-time status of their service to their customers.
However, when you're in the heat of an incident, it can be challenging to remember to update it. Where did we put the login, who has the right permissions, where do I go to update it? These are things we've been asked about time and time again by customers.
To help make things easier, we've brought one of the most popular status page providers out there - statuspage.io - right into your incident channels. Type /incident statuspage
to notify customers of new incidents, provide updates to ongoing ones, and to let them know when the impact is over.
To get started using this for your incidents, head over to our shiny new documentation which takes you through how to install and use it.
Statuspage has a great feature inbuilt that allows you to pre-fill the impact, update text, and affected components. This makes it less stressful when figuring out what to write. We don't currently support this, but we hope to have it out in a couple of weeks. If this is something you'd like to see, let us know!.
We've been on a bit of a roll recently when it comes to adding integrations. Over the past month, we've added Opsgenie, JIRA, GitHub and Linear. Well now, Clubhouse gets to join the club!
You can use our Clubhouse integration to export actions from your incidents, to aid prioritisation, and make follow-up a breeze. When you've exported your actions, we'll continue to track and sync actions into incident.io, and link back to the incident from any issues we create, in case you need to reference it later.
One of the things we knew wouldn't last long was "open/closed" modelling of incident states.
That's not how any incident we've ever been part of has worked and although it got us off the ground, we've been embarassed about it ever since.
Your feedback told us the same thing, and we decided it was high time we fixed it.
Incidents can now be in several states:
These states can be controlled via Slack, and the timestamps can be updated via the dashboard after the incident if any adjustments are needed later.
We think they're really useful for clearly communicating the progress of an incident to your team and stakeholders. Additionally, capturing this data lays the foundation for lots of interesting and useful insights, and we look forward to sharing those with you in the near future.
In the early days of building incident.io, we had a hunch that "storing a document" against an incident might be a useful thing, whether for post mortems, or for something else.
We've since had two bits of feedback repeatedly:
So, that's what we've done. The generic "document" link is no more, and in its place we now support a post mortem document explicitly. Great feedback, keep it coming!
Another mid-release update from us today. You can now export incident actions straight into GitHub Issues!
GitHub Issues provides a developer-friendly, lightweight way to track actions so it's no surprise it's been fairly high on the list of requests from customers.
Once exported, we'll continue to track and sync actions into incident.io, just like you can with our Jira and Linear integrations. We'll also link back to the incident from any issues we create, in case you need to reference it later. Happy GitHubbing!
We were a bit quieter last week because we've been heads down building and responding to feedback, so we're giving you a bumper update this week instead!
Hot on the heels of our PagerDuty integration, we got a lot of feedback from teams using Opsgenie, who felt a little left out.
We're delighted to say those days are over, and you can now escalate to your team via Opsgenie from right within your incident channels. Just like with PagerDuty, let us know who you need, and we'll handle the rest.
We've also got a few other nice integrations and improvements we'll be talking about more later this week.
A little mid-release update from us today. You can now export incident actions straight into Linear!
We're big fans, and selfishly we're pretty excited we can use this ourselves, as copying things over manually after incidents was getting a little tedious...
Once exported, we'll continue to track and sync actions into incident.io, and shortly we'll show you insights into how many actions you create, how many get followed up, and how quickly, too.
We'll also link back from tickets to the incident, in case you need to reference it later.
We want to meet you where you work, which means gluing together the tools you already use. We've got a long list of integrations planned, and Linear was getting a lot of demand, so we bumped it up the list.
Getting this kind of early access feedback to help us steer the product is really what drives us and makes building incident.io worth it, so thank you! If you have more integrations you'd like to see, drop us an email or let us know on twitter.
We bring you this update under the rays of some glorious spring sunshine 😎🌼.
One of the problems we hear time and time again is that following up on actions after an incident is harder than it should be. They get written down in slack or a post mortem document, maybe exported to an issue tracker, but more often than not, are then left to languish.
Without a clear link back to the incident that caused them, it's hard to track whether or not they get done, let alone see how long that takes and be able to report on it over time.
As a first step towards solving this, from today you can manage your actions via the incident homepage, and export them directly to Jira. We link them back to the incident that they were created from, and keep the state in sync for you.
Next up, we'll be building tools to help you make sure they get done, and surface insights so you can truly understand how quickly that's happening.
We've started with Jira, but Linear and others may follow soon. If you're keen to see that happen, let us know!
/incident help
, help will be returned in a modal that only you can see."Resolving our incidents is only half the story"
That's what you keep telling us, and we couldn't agree more.
To really get the value from your incidents, you need to follow up effectively. Right now, we're focussed on building out the tools to help you do just that. We're starting with extracting the insights and learnings which, let's face it, you've already paid for. We want to help you get your money's worth.
A common solution we've seen (and used ourselves!) is what we at incident.io HQ refer to as "The Google Sheet of Doom". Often this is a manually compiled spreadsheet, containing some high level numbers on things like duration, severity and impact, maybe with some graphs and Pivot Tables thrown in at a stretch.
We think we can do it better, and do it for you. Today we're taking the first step in that direction by launching the first version of Insights, available for all incident.io early access customers.
We love our bot. It takes rote work off our hands, and let us focus on the things humans are great at. However, it's important that the incident channel remains clean and easy to scan, and it were getting a little loud for our tastes.
We started the week deep in thought, coming up with design principles for how the bot should interact. We'll be writing about these soon - stay tuned to our blog for more.
Armed with these principles, we built them into the product. The bot should be noticeably quieter. Let us know what you think!
Do you ever find yourself rummaging through an incident channel for a dashboard someone shared, a runbook the team referred to earlier, or that documentation you were reading a few minutes ago?
We do — all the time.
From today, you can just type /incident links
and we'll scurry off and find them all for you. Let us know what you think!
It's often the case that you reach the boundaries of your knowledge during an incident, and you need that special someone to help you out. We've all been there. However, getting help can be hit and miss: tagging people in Slack, or trying in vain to blow the dust off your PagerDuty login.
Fear no more - you can now page users and rotations directly in Slack via /incident escalate. We'll stream notifications and acknowledgements into the channel as they happen so you know who's on their way.
We've launched with PagerDuty to get ourselves comfortable with the flow, but we'll be launching Opsgenie and VictorOps soon!
We've been getting tonnes of great feedback from our early access customers this week. We also said hello to 4 new ones. 👋🏼
Incidents can be fairly stressful, so the fact so many of you not only fixed your own bugs, but also helped us find and fix a few of ours is something we're incredibly grateful for.
We've been spending a bit of time thinking about how we can best share some of the longer term thinking we've been doing on what's coming next. This week we're really excited to launch our incident.io product roadmap, which lets you get a view into what we're up to.
There are upsides and downsides to sharing a product roadmap publicly, but we believe pretty strongly that:
We'll keep on shipping plenty of changes, improvements and fixes every week, but we'll use this as a way to keep you updated on some of the "big stuff" as it takes shape — enjoy!
This week was a particularly special week, as we onboarded our first 4 customers. 🎉
As such, we focused our time on helping them get onboarded, talking to them, and figuring out what we should build next.
We launched last week, and it went quite a bit better than we expected!
We've been inundated with support, feedback, potential customers and ideas for what to focus on next. However, pride comes before a fall, so we're being careful to keep focussed on talking to customers and building.
We spent the week putting together the final polish on the website before we went live. We've also been preparing the product ready to put it in the hands of early access customers, so we've been dotting the i's, crossing the t's and wrangling the experience to make it "just so".