Changelog

New kids on the blog

On top of the usual collection product improvements, this week we've also shipped our new blog. The old one felt a little uninspiring, and since we've got lots we want to share, we thought a little overhaul was a good investment in time. To mark the occasion we shared the first in our interview series, this time featuring Colm Doyle from Slack.

We've had a few things competing for our time over the last few weeks, which has slowed the pace of change a little. Next week's looking pretty clear though, with plenty of time to devote to product and engineering. Stay tuned!

What we shipped

• 🆕 We launched our lovely new blog and the first in our new incident interview series, this time with Colm Doyle from Slack. Kudos to Pete here!
• 🆕 We've made some changes to make incident summaries feel more connected to the incident. After you /incident update, we now prompt you to check that the overall summary still makes sense. A well written summary can go a long way to keep folks on the same page.
• 👷🏽‍♀️ We've been working with an external company to carry our a penetration test against our app and infrastructure. We're confident in our security setup, but it's great having someone try to find any holes. So far, things are looking great 🔒
• 👷🏽‍♀️ We've added some debugging tools to our codebase to help us find and fix things like memory leaks in future. As they say, failure to prepare, is preparing to fail.
• 👷🏽‍♀️ We used to perform a round trip to Slack on one of our identity APIs, which meant we coupled the success of that call to Slack. Given we were getting data that's unlikely to change often, we've moved it into cache.
• 💅 We used to nudge you to set the 'incident lead' even if you'd changed what the role was called. Now, if you call it a commander, we'll call it a commander 🤝
• 💅 We've shorted the timestamp we use in postmortems. Nobody needs millisecond precision here.
• 💅 We used to nudge you about the incident being quiet, even if you were in a monitoring state. When you're monitoring it's not uncommon for the channel to go a little quiet so this wasn't particularly helpful. Longer term, we'll make all of our nudges customisable, but for now we've restricted this nudge to investigating and fixing stages of the incident when we expect activity in a channel.
• 💅 We no longer show inactive or removed users in dropdowns, since assigning an action to someone who longer works for you probably isn't a wise move.
• 🐛 We fixed a bug that occasionally meant images wouldn't show up in the incident timeline. Long story short, when you loaded the incident homepage we'd generated a url for the image which would be valid for 10 minutes, and then cached the linked timeline item for 15 minutes. In those last 5 minutes, the image url had expired, but we still served the timeline with it. All fixed now though, so pin away!
• 🐛 Postmortem exports weren't working on Safari, but now they are!
• 🐛 We fixed a few minor issues with our Jira integration, and have some bigger changes on the way soon.
• 🐛 If you reopened an incident, there was a chance the incident homepage would stop working. Not anymore though! Thanks to RD Station for their help getting to the bottom of this 🙏

All onboard!

Over the last week we've been busy making it even easier for your whole organisation to onboard with incident.io. Whether it's simple step-by-step instructions for the installer, or helpful messages to folks after they've participated in their first incident, we've got you covered.

What we shipped

• 🆕 We've overhauled our sign-up and installation process. You'll now be taken through a full onboarding process in preparation for your first incident.
• 💅 We now handle redirects during login, so when you click that link to an incident, you'll actually get there.
• 💅 After providing an update with /inc update, we'll now prompt you to check the incident summary still makes sense. A few minutes getting everyone on the same page can save hours later on.
• 💅 It's possible to close a Statuspage.io incident without marking your components as operational. We now smooth that rough edge by prompting you to resolve them all together.
• 👷🏽‍♀️ We shared a blog post about a Postgres feature that caused some incident numbers to jump by 32. If edge cases in databases are your bag then check it out!
• 👷🏽‍♀️ We got caught out by a Slack workspace where channel creation was restricted to admins. We now handle that case to avoid half-complete installs.
• 👷🏽‍♀️ We've made improvements to our logging setup, so it's easier to pinpoint bugs when they (very rarely 😉) pop up.
• 🐛 We've fixed a problem that would sometimes prevent pinned Slack messages with images from showing up in the timeline. All fun and games until caching gets involved, but we've fixed this now.
• 🐛 When you closed an incident via the /inc update command whilst also changing the severity, we'd show the old severity in the closure modal. Now, thanks to a report from the folks at Ziglu, we show the correct one.
• 👷🏽‍♀️ We used to restrict Slack channel names to simple characters and numbers, despite Slack supporting diacritics. We now support everything Slack does, which is àwęšõmê!

All the small things

Instead of doing just one thing last week, we made lots of smaller improvements to the product. We also got a lot of internal admin off our plate to keep the paperwork gods happy and keep the wheels turning.

We won't bore you with the paperwork, but we hope you enjoy the new features and improvements!

What we shipped

• 🆕✨ Better incident declarations. We now show you a 'jump off' modal after you declare an incident (pictured). Might just be us, but we were finding it hard to quickly find incident channels after declaring an incident. This makes it a lot easier.
• 🆕✨ Automated incident closure. To keep your data nice and tidy, we'll automatically close incidents for you that haven't seen any activity for a time period you specify. Enable it in your settings to try it out!
• 🆕✨ Nudges to update your Statuspage. If you've created a Statuspage incident for your incident, but haven't updated it for 30 minutes or more, we'll send you a nudge to remember to do that. Now it's easier to keep your customers in the loop.
• 💅 We've unified the severity, status and update commands behind the update command. Fewer commands to remember makes the product simpler. However, using /incident status will still work just fine.
• 💅 We added support for Microsoft Teams as a video conferencing provider. If you paste a Teams link into an incident channel, we'll now prompt you to to add it as the call link.
• 👷🏽‍♀️ We added a vulnerability disclosure policy to our website. If you spot any security issues, please follow the instructions provided there.
• 👷🏽‍♀️ We made a number of resiliency improvements, mostly focussed on our job processing infrastructure and our statuspage integration. Hopefully nothing too noticable, but it helps our team sleep better at night knowing it's all shipshape.
• 🐛 Organisations with 5 or more severities would have all of their severities appear in a delightful mauve on the Insights graph. Whilst that was quite nice, it wasn't intended, so we fixed it 💜 (thanks to Duffel for reporting!)
• 🐛 The dates in our insight graphs were sometimes appearing offset by a month. All fixed, no more time travelling (thanks to Duffel!)

Quick re-actions

Incidents are a team sport, and to help everyone get involved we built incident actions. With a quick /inc action in Slack, anyone can view, create, and pick up actions, making it easy for folks to self organise around whatever needs to be done to get things back on track.

We got some feedback that the actions interface in Slack could be a little clearer though, for example the layout was a little cluttered and it wasn't easy to see who created what. We've now refined the design and tweaked what we display so it's even easier to get a clear picture of what's going on.

Got thoughts on this or any other part of incident.io? Jump into our community Slack and let us know!

What we shipped

• 🆕 We want the trial experience to be super low friction, which is why we don't ask for a credit card up front. To save us personally nudging folks at the end of their trial, we now show you when you're close to the end, or over your trial period.
• 💅 The actions modal has been given a little overhaul. We've simplified the layout so it's easier to see what's going on, and we now show who created open items so you can ask them for more details if needed.
• 💅 If you typed /incident help from outside an incident channel, we used to open the create incident modal with an incident title of "help". You were probably after some actual help, so we now show you modal with some tips. Thanks 100/x for this feedback!
• 💅 When you joined an in-progress incident, we used prepend the summary with "All systems go in here!". On reflection, this wasn't the clearest language when your systems often weren't going. We're a little less ambiguous now.
• 💅 We now support typeahead search in the Escalations modal. You can now filter to find the person or escalation policy you're after.
• 💅 We used to show PagerDuty users and escalation policies in a non-deterministic order. That wasn't especially nice, so we now show them in a consistent order.
• 💅 We new prompt you to provide an incident update if you haven't done so for a while. Keeping folks on the same page is important, and we know how easy it is to forget.
• 💅 We've spruced up our login flow, so it's no longer just a lonely login button on a blank screen(!). It's had some design applied, and we explain what people in your organisation will be asked for and why.
• Links to the incident homepage now automatically embed the teamID wherever we can, so logging in with slack auto-fills to the right workspace. This helps people avoid accidentally signing into the wrong one, something we've seen bite a few of you!
• 👷🏽‍♀️ We've improved how we poll PagerDuty for updates. This means we need to do less work, and we're also a little less trigger-happy with the PagerDuty API. It's nice to be considerate.
• 👷🏽‍♀️ We've made some reliability improvements in the way we run concurrent code in the app. We'll spare you the nerdy details, but things are much more resilient as a result.
• 🐛 If you had more than 100 components configured in Statuspage.io, we'd fail to render them in our Slack modal. All sorted now; create components to your heart's content!
• 🐛 If you pinned a Slack message from a bot, we'd fail to put add it to the incident timeline. All sorted now - pin away!
• 🐛 We use content security policies in our front end, to keep things safe and secure. We were being slightly too secure with our shiny new Intercom chat, which meant some functionality was broken. We've fixed that - feel free to give it a try and say hello 👋

Are you talking to me?

We're fanatical about great support here about incident.io. Every customer gets a shared Slack channel to make it easy to ask questions and give feedback. We've now made that even easier by adding Intercom to incident.io.

Whether it's a question about how to add a custom incident role, or feedback on an improvement we could make, you can now send that to us in a couple of clicks. Click the button in the bottom right hand corner of incident.io to get started.

Behind the curtain, Chris, Pete and Stephen will be diligently answering your questions, fielding your feedback and blushing at your compliments. Looking forward to hearing from you!

What we shipped

• 🆕 We've added two new Looms to help you learn how to set up and use incident.io. We send now send you a link to a video tutorial when you install incident.io, and when you create your first incident. We love you, Loom!
• 💅 When you add an update via /incident update, we now default your next update to 30 minutes, as part of encouraging better defaults.
• 💅 We've sped up every web request by approximately 200ms by introducing caches in the right places. 🚀
• 💅 When you join the channel for a closed incident, we'll send you a message with some useful links to the homepage, and to your debrief.
• 👷🏽‍♀️ We spent a lot of last week working on fundamentals to make our application scale better. We can't give you all the juicy details, but we promise we're working hard!
• 🐛 After releasing incident updates last week, we discovered that we were still reminding you for updates after the incident was closed. That was silly, and we don't do that anymore.
• 🐛 We send you a nice message to get you up to speed when you join a live incident channel. In some circumstances, we'd send the summary twice as part of that. Regardless of how beautifully written your summaries are, we only want one copy to appear.

A Quick(er) Update

Good incident response is about good communication. In incidents, it's particularly helpful to provide a small, structured update to the organisation on a regular basis. This keeps everyone on the same page, and helps resolve worries about when the next update will be coming, too.

We're always looking for ways to make good communication easier, and a few customers have mentioned that using seperate commands to udpate the severity, status and summary often felt a bit harder than it needed to be. You might not remember to update the status, or forget to update the summary after new information came to light — we certainly did!

We think all these fields work better together, so last week we set to work on incident updates!.

Updates are an easy way to provide a small, structured update to the rest of the organisation on a regular basis. We tie together existing fields like Status and Severity, with a new field allowing you to share any other context. Additionally, if you set a reminder for when you're expecting to send the next one, we'll remind you at just the right time ⏰

Hopefully this makes internal communications a lot easier, and reduces some of the burden when it comes to communicating regularly. Give updates a spin in your next incident, and let us know what you think!

What we shipped

• 🆕 We added grouping to the /incident command, to make it easier to understand what you need to do in the middle of an incident. You can now Communicate, Coordinate, Review and Help to your heart's content.
• 💅 We changed the text when asking for feedback to make it clear that the feedback is for your organisation to improve at incident response, not feedback about incident.io. As much as we would love that feedback, we think it's more important for you to share it amongst your team.
• 💅 Adding a postmortem document from the web will now update the incident channel.
• 💅 Statuspage updates are now asynchronous when you post them. This means that we're more resilient to failure if Statuspage is down, or a little slow.
• 👷🏽‍♀️ We spent a bunch of time improving the reliability and correctness of our Statuspage integration. It's now much faster, and more failure-tolerant. Thanks for the feedback, Render!
• 👷🏽‍♀️ Announcing and updating the incident in your #incidents channel now happens asynchronously. This allows us to be more resilient to failure, and provides a 300ms speed up for every command 🚀
• 🐛 When you created an incident from a message, we weren't attaching the name of the channel you created it from in our response to you in certain circumstances. That was silly, and we're sorry. It's now fixed.

Start with Wye

It's a been a while since we last posted one of these, and despite the radio silence we've been busy behind the scenes at incident.io HQ. Firstly, we now actually have a HQ, and it's in an old firestation - very fitting. Pete and Stephen have left their full-time jobs and are now working 100% on incident.io.

This week we've decamped to an eco-lodge in the Wye Valley to plan for the next few months. We've got hundreds of ideas and a mountain of valuable feedback, so we're taking a week to refine it all into something amazing.

No more breaks now. Expect weekly changelogs to be weekly. All systems go.

Thanks for the feedback!

We received some lovely feedback from customers over the past week — this kind of thing makes our day, thanks folks!

We at @Farewill started working with the fantastic @incident_io team earlier this year, and honestly it makes life SO much easier to have seamless tooling when everything is on fire. Watching our engineers working in the incident channel to check how we're affected is just 🤩

— Sally Lait (@sallylait) June 8, 2021

Check out @incident_io to manage incidents within slack. Awesome workflow and all the automation you ever wanted to bring more structure and fun into a potentially stressful situation. We'll start using it for every alert @giantswarm now.

— Timo Derstappen (@teemow) June 12, 2021

Super awesome demo of @incident_io from @sjwhitworth earlier. Really excited about what the team is doing to switch up incident management by bringing together tools we use & love into a streamlined workflow🔥🔥

— Nick Coates (@nickcoatesuk) June 10, 2021

What we shipped

• 💅 We've stopped Slack unfurling links when we post from @incident. We share all the context you need, so pulling it from an external link just cluttered the channel.
• 👷🏽‍♀️ We've been working on some big resiliency improvements; splitting parts of the app into distinct components to make us more reliable and tolerant of failure.
• 👷🏽‍♀️ We've been improving the state of our build and deploy pipelines. We can now test our code, and deploy it through to production in ~5 minutes - handy as we start ramping up on shipping ⚡
• 👷🏽‍♀️ We've added a number of alerts so we'll know quicker when things go wrong, and specifically what's up.
• 🐛 We fixed a bug with our statuspage integration which was causing errors if you opted to tweet the status update too. Thanks for the report Render!
• 🐛 We use idempotency keys to prevent duplicate items being added to the incident timeline, and one of our keys included the current time. Not a great choice, clearly, but so we're now using something reproducible.
• 🐛 We fixed a bug that showed the incident reporter as the person who generated the postmortem document. Now the correct person gets all the credit.

What's up, doc?

We now have a number of customers using us daily to respond to incidents, and we're seeing more of you leaning into improving your post-incident followup. This is an area where until now we've been heavily opinionated, without much room for customisation - something we always knew we'd need to change.

We want to make it easy to imprint your process directly into incident.io - we want you and your team to feel like this is your tool, not someone else's.

We've always been able to automatically generate incident documents (also often referred to as post-mortems) and timelines automatically, but now you can now customise the format to your heart's content.

What we shipped

• 🆕 Assigning someone a role in an incident automatically invites them to the channel. Thanks for the polish suggestion, Upvest!
• 🆕 We added lots of new logos to our website. Hello WorkOS, Gitpod, Livestorm and Prolific!
• 💅 We added typeahead support to all places where you can select a user in Slack. This means we can support much larger organisations, and make it easier to search.
• 💅 We shortened the message that we send when an incident is closed. It was a little long and in-your-face.
• 💅 When creating an action in the dashboard, we now sort the list of users that you can assign the action to by name to make it far easier to scan. Thanks for the prod, Truelayer!
• 💅 We flipped the ordering of ratings when giving feedback so it goes lowest to highest, instead of the reverse. It felt more natural, you know?
• 💅 We now only show you a nudge to customise your roles if you haven't already done so. You're clearly an expert, so we don't need to waste your time with the basics.
• 💅 The incident lead/commander is now always shown first when interacting with roles in modals or settings. Because they're important like that.
• 👷🏽‍♀️ We totally finished our migration to slash commands. Conversational UI, be no more!
• 👷🏽‍♀️ We added company information to our receipts and invoices. We're growing up. Thanks for the prod, Upvest!
• 🐛 If you tried to close an incident twice, in some circumstances, we'd post two messages. We now handle this idempotently.
• 🐛 We found a particular edge case where changing the incident severity via a modal after an incident was closed would remove the closed_at timestamp that we use for tracking. That was silly. It now doesn't do that. Thanks for helping us spot this one, Farewill!
• 🐛 We found a rare issue where we'd fetch the wrong message from Slack. This happens when we try and fetch a message that has been deleted. If it has, Slack will return the message immediately before the deleted message - which wasn't the message of interest. We've now fixed that! Sorry for the phantom bugs, Upvest and Bud!

They see me role-in

In incident.io, roles are a first class citizen. We think of them as 'hats that you can wear' during an incident. We use them to make it clear:

• What the responsibilities are of a given role (e.g. the Communications Lead is responsible for internal and external communications)
• Who is currently playing that role (e.g. Pete)

Until now, we only supported a single role - the Incident Lead. We knew this was limiting, and our customers agreed, so we fixed it.

You can now add and customise your own roles to fit how you like to respond to incidents, attaching a description and instructions on how to play the role. Roles can be assigned and re-assigned during incidents by using /incident roles.

When someone is assigned to a role, we'll privately send instructions to the assignee to make it clear what they need to do, and help them get up to speed.

Head over to the Help Centre to learn how to use them, or join us in Slack to let us know what you think!

What we shipped

• 🆕 If you /zoom in an incident channel, we'll prompt you to add it as the call for the incident. Like magic.
• 🆕 If a user joins an incident channel, we'll now automatically acknowledge any open PagerDuty escalations they are assigned to. This’ll prevent those accidental escalations to others when you jump into incident mode, and saves you precious time and headspace.
• 🆕 We have a small-but-lovely Community where we share sneak previews of features, and get open feedback on the product. We now invite you to join it after you've closed your first real incident.
• 💅 If you make updates to the incident as you close it, we used to splurge a lot of additional updates into the channel, in addition to the closing message. We're now a lot quieter. Thanks for the prod, GoCardless!
• 👷🏽‍♀️ As part of our migration to slash commands, we removed all references to @incident in our codebase.
• 🐛 If you tried to close an incident when you'd already archived the channel, we would barf. Now we don't. Thanks, Loom!
• 🐛 In some circumstances, we weren't announcing when actions were completed. We now do.
• 🐛 Occasionally, we wouldn't update the incident announcement channel post after some updates via a slash command. We now ensure that we always update it, regardless of the command you use. Better to be safe than sorry.

Slash. Saviour of the universe

When we started incident.io, we felt passionately that we should replicate the feel of human communication during an incident. Need to escalate to someone? Just talk to us with @incident escalate and we'll guide you through it. The drawback? We made a lot of noise in your incident channel 🙉

We've listened to your feedback, and today we're making the move to Slack's slash commands. Whilst it's just a single character, moving from '@' to '/' has a big impact. When you /incident escalate it makes no noise in a channel, and as an added bonus we can take you straight to the right place. It's a win-win!

As if that's not enough, we're also introducing the Incident Home. Can't remember the command for the status page? Just type /incident (or /inc if you're in a hurry) and we'll help you get to the right place.

What we shipped

• 🆕 When you pin images in an incident channel, we'll add them to the timeline. Worth a thousand words.
• 🆕 We added lots of new customer logos to our homepage. Hello, new friends! 👋🏼
• 💅 When you export actions to your issue tracker, we'll now show the link within the actions modal in Slack.
• 💅 We've added tooltips to the participants on the dashboard. Now you can see exactly what we mean by a 'Key Contact'.
• 💅 We've stopped sending incident action updates to the channel after the incident is closed. These updates aren't super helpful, and draw attention the channel when it's not needed.
• 🐛 If you split your severity definitions over multiple lines, we didn't display the text nicely in Slack. Now we do! Thanks Duffel!
• 🐛 We found an incident in the future on the Insights page. We can't predict future incidents (yet) — it was just a timezone issue.
• 🐛 If you swapped the names of your incident severities, we didn't always update them when you hit save. If it's not DNS, it's overly zealous database constraints.

Onward and Upward

Most of the energy around incidents is, quite rightly, focussed on coordination and solving the problem, but there's a tonne of value in what happens afterwards, too.

Whether it's to spot topics for a postmortem, or learn which parts of your incident response could be improved, just asking "how did we do" is a powerful tool.

Often we forget to ask, and it involves a lot of additional admin to follow up and collect responses, but from today, incident.io will handle it for you.

We'll automatically check in at the end of an incident and ask everyone how it went, with a few specific questions, as well as some more general prompts.

This is just a first pass, but we've already seen customers using it to gather ideas for improving their incident response. Let us know what you think!

Welcome to the party

Our main focus this week has been making sure it's easy to learn what incident.io is, and how it works. We've spent a lot of time working on great tooling for incident leads. However, by definition, there's only a single incident lead but many participants and observers. We want to make sure you have a great experience too!

As part of that, we've polished the messages that you get when joining an incident channel to make them clearer and more concise. Additionally, you can indicate whether you're there to help (we'll announce it in the channel if you are), there to observe, or if you want a quick introduction to incident.io.

We'll spend the rest of this week focusing on improvements to onboarding: look out for some new features coming out over the next few days!

What we shipped

• 🆕 Ever felt like your incident channel names didn't quite reflect who you were? I feel you. Well, incident channel name prefixes are now customisable: take your pick.
• 🆕 Want to write an update on your Statuspage without telling the world about it? You can now optionally set whether to deliver notifications to customers that have subscribed - previously, it was mandatory.
• 🆕 As part of our gradual but continual march to slash commands, we now support lead, rename, doc and call. Give it a try with /incident lead me. Only a few left to go now..
• 🆕 Until this point, we didn't have a Statuspage. As a business that aims to make it easier to respond when things go wrong, that was an omission. You can now find us at status.incident.io. We, of course, will be using our Statuspage integration to update it.
• 🆕 We added lots of new customer logos to our homepage. Hello, new friends! 👋🏼
• 💅 Statuspage components now have their component group included in their name, if they have one.
• 💅 If you added an action and assigned it to a user using /incident action, we'd send two separate updates into the channel. We've combined that into one update. Thanks for the feedback, Upvest!
• 💅 While we're on actions, we no longer post updates into the channel if you add new ones or update them after an incident is closed, keeping your channel history a little cleaner, once an incident is closed.
• 💅 The ID and name of the incident now appear in the browser title when viewing the incident homepage. Nice idea, Upvest and FACEIT! 🙏🏽
• 💅 When incident channels get archived, some features of the app aren't available. We used to tell you this, but not explain why. Now we do, avoiding a poor "computer says no" experience!
• 🐛 We used to return Statuspage component groups as a component that you could say was affected. That was a little strange, so we don't do that anymore. Thanks to Bud for noticing this one!
• 🐛 If you /incident summary Hello world, we used to discard Hello world. That now forms part of the update that appears in the modal. Sorry about that.
• 🐛 In the document export of the timeline, pinned messages would appear at the timestamp that you pinned thme, not the timestamp that the message occured at. That was silly, and it's now fixed. Thanks FACEIT!
• 🐛 If you had more than 100 PagerDuty services, we'd error when you tried to escalate. We now perform typeahead search, so it's easier for you to find the service of interest.

Testing, testing, 123

Our customers want a way to train people on incident.io, without unnecessarily worrying others in the organisation that things are going sideways or messing up their lovely incident library and statistics. Until now, you had to use normal incidents to do this. That's no good.

Well, enter test incidents! Test incidents are exactly the same as normal incidents, except:

• We won't announce them to the organisation in #incidents
• They're excluded from your incident library or statistics
• We disable some integrations like Statuspage, to avoid embarrassment

We think that they're best used for:

• Training your organisation to use incident.io
• Testing new features out
• Practicing your incident response on game days

Kick off an incident by /incident test and let us know what you think!

All aboard!

First impressions are important. We've seen more and more people using incident.io recently, which has been fun to see, so we've used this influx of new faces as a prompt to think about what we could do better here.

We're currently focussing on onboarding and adoption to make it easier for new users of incident.io to get up to speed, whether that's when you install us, or the first time you're pulled into an incident.

We'll have more to show next week, but we've already made a start, so you'll probably notice a few changes. If you spot them, let us know what you think!

What we shipped

• 🆕 When you install incident.io to your workspace, you now get a much clearer message that tells you what we've done, and what you should do next.
• 🆕 When you join the #incidents channel for the first time, we now introduce ourselves, and tell you what the channel is for. We do this privately, of course.
• 🆕 If you want to automatically have a call attached to your incident at kickoff, you can! Some folks have a consistent call always at the ready, and now we'll automatically add it to your incident for you (thanks to Bud for suggesting this!)
• 🆕 Fed up of typing /incident to kick off an incident? Too many characters and you're falling asleep by the end? Well, you can now type /inc to save you time, and effort.
• 🆕 Keen readers of The Changelog will have noticed we're starting to experiment with slash commands in incident channels. This week, we added support for a bunch of commands, including close, recap, statuspage and escalate. Try /incident recap for a quick spin!
• 💅 If you escalate using Opsgenie, we'll now stream updates into the channel for you.
• 💅 If you created an incident with /incident The website is down, we used to ignore The website is down and make you type it again. We now automatically populate that into the modal, to save energy for your precious little fingers. Thanks Ravelin, Bud and many others for this feedback!
• 👷🏽‍♀️ Our backend application is currently one large program, and we hope it stays that way. We're making some improvements to how we structure it to improve speed of development, reliability and correctness.
• 👷🏽‍♀️ We invested a bit of time in our build and deployment pipeline - getting it down from 10m to a far healthier 2m, every second counts!
• 🐛 If you logged in to app.incident.io on multiple Slack workspaces, you'd sometimes see an error when we thought you'd installed the app in a workspace, when you had not. We've now fixed that!
• 🐛 The biggest lie that software engineers tell themselves is that "we'll do pagination later". We uttered these words when building our PagerDuty integration to show you the list of services, and unfortunately, you were limited to 25. We implemented pagination, and now there are no limits!

Getting on the same page

In 2021, most companies have a status page set up to make it easier to publicly communicate the real-time status of their service to their customers.

However, when you're in the heat of an incident, it can be challenging to remember to update it. Where did we put the login, who has the right permissions, where do I go to update it? These are things we've been asked about time and time again by customers.

To help make things easier, we've brought one of the most popular status page providers out there - statuspage.io - right into your incident channels. Type /incident statuspage to notify customers of new incidents, provide updates to ongoing ones, and to let them know when the impact is over.

To get started using this for your incidents, head over to our shiny new documentation which takes you through how to install and use it.

Statuspage has a great feature inbuilt that allows you to pre-fill the impact, update text, and affected components. This makes it less stressful when figuring out what to write. We don't currently support this, but we hope to have it out in a couple of weeks. If this is something you'd like to see, let us know!.

What we shipped

• 🆕 We built a beautiful Help Centre. You can now learn all about what incident.io is, and how to use it.
• 🆕 We created the incident.io Community - a Slack workspace for us, and our lovely customers to hang out in. We'll use this to share early work, get feedback, and discuss interesting topics as a group.
• 💅 When you close an incident, we'll ask you to assign an owner to the postmortem. Every little nudge helps (thanks to Bud for this one!).
• 💅 You can hide a PagerDuty escalation policy or service from appearing in incident.io by adding the #incident-io-ignore tag anywhere in the description. This helps if you have test rotations, or a list of Very Important People that you'd never like to page (Bud again 🙌)
• 💅 We were a little short-termist when showing you your insights, defaulting to the last 7 days. We now show you 30! What will you do with your extra 23 days? (thanks for the feedback Upvest!)
• 👷🏽‍♀️ We're prototyping using slash commands to interact with the bot. This provides us with a lot more powers out of the box. We're seeing what it feels like first, before rolling it out more widely. Watch this space! 👀
• 👷🏽‍♀️ On the very rare occassion that we crashed, we'd return the default Heroku 'app crashed' page. We now return something more beautiful, and witty. We hope you never see it 🙈.
• 🐛 When adding an Opsgenie integration, we weren't as thorough as we liked with checking that you were on the right tier to have API access enabled (grumble, grumble). We now check that at integration time - sorry for the hassle, Ravelin!

Join the Club

We've been on a bit of a roll recently when it comes to adding integrations. Over the past month, we've added Opsgenie, JIRA, GitHub and Linear. Well now, Clubhouse gets to join the club!

You can use our Clubhouse integration to export actions from your incidents, to aid prioritisation, and make follow-up a breeze. When you've exported your actions, we'll continue to track and sync actions into incident.io, and link back to the incident from any issues we create, in case you need to reference it later.

Stating the Obvious

One of the things we knew wouldn't last long was "open/closed" modelling of incident states.

That's not how any incident we've ever been part of has worked and although it got us off the ground, we've been embarassed about it ever since.

Your feedback told us the same thing, and we decided it was high time we fixed it.

Incidents can now be in several states:

• As soon as you kick off an incident, you'll be Investigating it
• Once you've figured out what's wrong, you'll be Fixing it.
• Once you think you've put a fix in place, you'll be Monitoring it.
• Once it's all over, the incident will be Closed.

These states can be controlled via Slack, and the timestamps can be updated via the web UI after the incident if any adjustments are needed later.

We think they're really useful for clearly communicating the progress of an incident to your team and stakeholders. Additionally, capturing this data lays the foundation for lots of interesting and useful insights, and we look forward to sharing those with you in the near future.

What we shipped

• 🆕 You can add postmortem links explicitly now
• 🆕 You can create actions from the dashboard, so if you'd rather follow up there rather than in Slack, you can
• 💅 Some of our Slack notifications were incredibly unhelpful and just said "content cannot be displayed". We found a way to make them displayable. Thanks to the fine folks at Ravelin for spotting that one.
• 💅 We now use the issue reference in the "open in X" links for Jira, Linear, GitHub issues etc (thanks, Future!)
• 💅 Sometimes you just want to export a follow-up action without assigning it to anyone. Now you can.
• 💅 We now announce when actions are unassigned, with the option for someone else to pick them up.
• 👷🏽‍♀️ We're a little smarter at matching users in Jira to users in incident.io than we were last week - every little helps!
• 🐛 We've added action completion times and links to external issue trackers to the generated postmortem documents (thanks GoCardless!)
• 🐛 When you completed actions via the dashboard, the completed_at wasn't updating
• 🐛 For organisations with lots and lots of Jira users, we weren't returning them all when assigning followup actions. Now everyone can pitch in, which feels much fairer (thanks Future!)
• 🐛 We more accurately reflect the "completion time" for actions updated via Jira and the web UI
• 🐛 If you talked to us in a thread, we responded in the main channel to let you know that doesn't work, which was weird. Now we keep the conversation threaded (thanks Ravelin!)

Purposeful Postmortems

In the early days of building incident.io, we had a hunch that "storing a document" against an incident might be a useful thing, whether for post mortems, or for something else.

We've since had two bits of feedback repeatedly:

1. We only use to store the post mortem document
2. We'd like that to be more clearly reflected on our incident homepage

So, that's what we've done. The generic "document" link is no more, and in its place we now support a post mortem document explicitly. Great feedback, keep it coming!

Midweek issue, GitHub edition

Another mid-release update from us today. You can now export incident actions straight into GitHub Issues!

GitHub Issues provides a developer-friendly, lightweight way to track actions so it's no surprise it's been fairly high on the list of requests from customers.

Once exported, we'll continue to track and sync actions into incident.io, just like you can with our Jira and Linear integrations. We'll also link back to the incident from any issues we create, in case you need to reference it later. Happy GitHubbing!

Wish granted

We were a bit quieter last week because we've been heads down building and responding to feedback, so we're giving you a bumper update this week instead!

Hot on the heels of our PagerDuty integration, we got a lot of feedback from teams using Opsgenie, who felt a little left out.

We're delighted to say those days are over, and you can now escalate to your team via Opsgenie from right within your incident channels. Just like with PagerDuty, let us know who you need, and we'll handle the rest.

We've also got a few other nice integrations and improvements we'll be talking about more later this week.

What we shipped

• 🆕 Although it's only been out for a month, we beautified our escalation flow. It's easier to understand, and you can now page multiple people at the same time. Go forth and disrupt their sleep.
• 🆕 We added some of our early customers to our homepage. We feel very lucky to have you all 😻
• 💅 If you export an action to Jira, we'll attempt to match the Jira assignee to an incident.io user, so it's clear who's doing what. We'll also sync the time that the action was completed back to incident.io.
• 💅 If you're lucky enough to have your message pinned as part of an incident timeline, your lovely face will appear next to it.
• 💅 If you export your actions to Linear, GitHub Issues or Jira, assignees are now optional.
• 💅 Previously, the push notifications that you received from us would say "This content can't be displayed". That wasn't very good. We now try and put something helpful in every Slack notification that is sent on our behalf.
• 💅 We now announce when actions have been unassigned (thanks Farewill!)
• 💅 If you got a command wrong whilst talking to the bot in a thread, we'd embarrass you by responding in the main channel. We've recognised the error of our ways, and now respond to in a thread instead (thanks Ravelin!)
• 👷🏽‍♀️ We re-wrote a lot of how actions work under the hood to make them more resilient, and well-tested.
• 👷🏽‍♀️ We've started sketching how we'll build Custom Attributes. We're excited for this one.
• 🐛 If you didn't have an incident in the past week, we'd assume that you'd never had an incident, and wouldn't show you any insights as a result. That was embarassing. That's now fixed.
• 🐛 We squashed a few small but pesky issues with our Jira integration, such as sporadic timeouts.
• 🐛 If you created an action that mentioned a Slack channel or user, it would look quite ugly in the dashboard. We've now cleaned that up, so it works how you'd expect.

Export to Linear

A little mid-release update from us today. You can now export incident actions straight into Linear!

We're big fans, and selfishly we're pretty excited we can use this ourselves, as copying things over manually after incidents was getting a little tedious...

Once exported, we'll continue to track and sync actions into incident.io, and shortly we'll show you insights into how many actions you create, how many get followed up, and how quickly, too.

We'll also link back from tickets to the incident, in case you need to reference it later.

We want to meet you where you work, which means gluing together the tools you already use. We've got a long list of integrations planned, and Linear was getting a lot of demand, so we bumped it up the list.

Getting this kind of early access feedback to help us steer the product is really what drives us and makes building incident.io worth it, so thank you! If you have more integrations you'd like to see, drop us an email or let us know on twitter.

Lights, camera, action

We bring you this update under the rays of some glorious spring sunshine 😎🌼.

One of the problems we hear time and time again is that following up on actions after an incident is harder than it should be. They get written down in slack or a post mortem document, maybe exported to an issue tracker, but more often than not, are then left to languish.

Without a clear link back to the incident that caused them, it's hard to track whether or not they get done, let alone see how long that takes and be able to report on it over time.

As a first step towards solving this, from today you can manage your actions via the incident homepage, and export them directly to Jira. We link them back to the incident that they were created from, and keep the state in sync for you.

Next up, we'll be building tools to help you make sure they get done, and surface insights so you can truly understand how quickly that's happening.

We've started with Jira, but Linear and others may follow soon. If you're keen to see that happen, let us know!

What we shipped

• 🆕 Money! You can now pay for incident.io through a shiny new billing page, and we'd very much appreciate it if you did.
• 🆕 Asking for help is now less noisy. If you mention /incident help, help will be returned in a modal that only you can see.
• 🆕 Actions are now exported to your postmortem document.
• 💅 When pinning a message to the timeline, we now include the name of the person that sent the original message, along with the person who pinned it. How lovely.
• 💅 We'll only send nudges when the channel appears to be active. This means we won't send repeated nudges in the channel, which makes us look a bit silly.
• 👷🏽‍♀️ We improved the healthchecks that happen before we deploy a change to production. Yay, resilience!
• 👷 We cleaned up a few unexpected frontend errors we were periodically seeing in production. They were small, and nobody noticed, but we knew, and we weren't having any of it.
• 🐛 If you escalated to the person that installed PagerDuty to incident.io, PagerDuty would automatically acknowledge the page, and not wake the person up. Unfortunately for the installer, they will now be woken up. Sorry, installer.
• 🐛 We got bitten by a classic "DivByZero" bug in our insights stats calculations. Easily missed, easily fixed 🙈
• 🐛 We weren't colouring your severities unless they matched the defaults, so if you used SEV1/SEV2/SEV3 they were all an uninspiring shade of grey in the dashboard. Now they're in technicolour, too!

Goodbye, Google Sheets

"Resolving our incidents is only half the story"

That's what you keep telling us, and we couldn't agree more.

To really get the value from your incidents, you need to follow up effectively. Right now, we're focussed on building out the tools to help you do just that. We're starting with extracting the insights and learnings which, let's face it, you've already paid for. We want to help you get your money's worth.

A common solution we've seen (and used ourselves!) is what we at incident.io HQ refer to as "The Google Sheet of Doom". Often this is a manually compiled spreadsheet, containing some high level numbers on things like duration, severity and impact, maybe with some graphs and Pivot Tables thrown in at a stretch.

We think we can do it better, and do it for you. Today we're taking the first step in that direction by launching the first version of Insights, available for all incident.io early access customers.

Quieten down, bot!

We love our bot. It takes rote work off our hands, and let us focus on the things humans are great at. However, it's important that the incident channel remains clean and easy to scan, and it were getting a little loud for our tastes.

We started the week deep in thought, coming up with design principles for how the bot should interact. We'll be writing about these soon - stay tuned to our blog for more.

Armed with these principles, we built them into the product. The bot should be noticeably quieter. Let us know what you think!

What we shipped

• 💅 Incident status is presented in a modal, rather than a thread.
• 💅 If it's clear that an action has been taken, the bot will ack with emoji rather than repeating what's happened.
• 💅 When creating an incident, the message we show to the reporter is smaller, with some handy buttons right there in front of you. Why type, when you can click!
• 💅 When we detect a call link and offer the button to set it, we'll now clean up after ourselves and remove the original message
• 🐛 When adding a call link to the incident after we suggested it to you, the button wasn't going away. Now it does!

The missing link

Do you ever find yourself rummaging through an incident channel for a dashboard someone shared, a runbook the team referred to earlier, or that documentation you were reading a few minutes ago?

We do — all the time.

From today, you can just type /incident links and we'll scurry off and find them all for you. Let us know what you think!

Well, that escalated quickly

It's often the case that you reach the boundaries of your knowledge during an incident, and you need that special someone to help you out. We've all been there. However, getting help can be hit and miss: tagging people in Slack, or trying in vain to blow the dust off your PagerDuty login.

Fear no more - you can now page users and rotations directly in Slack via /incident escalate. We'll stream notifications and acknowledgements into the channel as they happen so you know who's on their way.

We've launched with PagerDuty to get ourselves comfortable with the flow, but we'll be launching Opsgenie and VictorOps soon!

What we shipped

• 🆕 Actions are now reflected in the timeline, so you can see exactly when something was picked up, put down, or passed along.
• 💅 If we can't render items in your timeline due to permissions issues, we'll let you know and if it's fixable, we'll tell you how.
• 💅 The link to the Slack channel from the incident homepage now refers to it by name, rather than being generically worded.
• 💅 If you link a document to your incident, we'll show that on the incident homepage, too.
• 👷🏽‍♀️ We have to fetch lots of data to render your incident homepage. We now do more of that fetching in parallel to save you time. What will you do with your extra milliseconds?
• 🐛 Severities no longer fail to update in the channel topic if you update incidents from an edit modal
• 🐛 We no longer sometimes shortchanging users in incidents by incorrectly labelling them as "observers", when they were picking up actions, resolving issues and generally being good eggs.
• 🐛 Renaming channels to really, really, really long names no longer fails to update the post in the incident announcements channel.

Life on the open roadmap

We've been getting tonnes of great feedback from our early access customers this week. We also said hello to 4 new ones. 👋🏼

Incidents can be fairly stressful, so the fact so many of you not only fixed your own bugs, but also helped us find and fix a few of ours is something we're incredibly grateful for.

We've been spending a bit of time thinking about how we can best share some of the longer term thinking we've been doing on what's coming next. This week we're really excited to launch our incident.io product roadmap, which lets you get a view into what we're up to.

There are upsides and downsides to sharing a product roadmap publicly, but we believe pretty strongly that:

1. Unless there's a strong reason not to, we'd prefer to default to transparency.
2. We're building this product with you, our customers, and we'll work better together if you can see what we're up to.
3. The feedback is clear, and we've got high conviction in what our customers want, so we're happy to write it down and talk about it openly.

We'll keep on shipping plenty of changes, improvements and fixes every week, but we'll use this as a way to keep you updated on some of the "big stuff" as it takes shape — enjoy!

What we shipped

• 🆕 You told us not having your emoji in the timeline was practically criminal. We wholeheartedly agree. They now render right there in the timeline.
• 🆕 You told us our Slack topics weren't, well, topical. We've revamped them so they're much snappier, and show much more useful information.
• 🆕 You asked for more nudges, so we've added a few for updating the incident summary and updating the status page which pop up after ~30 minutes or so.
• 🆕 We've added whereby as a recognised call provider.
• 🆕 When you ask for a status update, we now also give you the option to edit the incident directly from the message.
• 💅 Incident timelines now show a lot more useful information, which is great, but you told us they were starting to feel a little squished. They now have much more room to shine.
• 💅 Lots more incident information is now available tthrough the web UI. Links to the Slack channel, any ongoing incident calls and more.
• 💅 Postmortem documents you generate automatically link back to the incident homepage.
• 👷🏽‍♀️ We launched a system to process events asynchronously (we use Google Pub/Sub, if you're interested). This means we can be more robust to failure, and automatically retry when things go south.
• 🐛 Our Content Security Policy headers were occasionally butting heads with the Avatar URLs Slack was returning. Your face should be smiling back at you in the nav bar once again.
• 🐛 Unpinned items no longer appear on the timeline.

Hello, new friends

This week was a particularly special week, as we onboarded our first 4 customers. 🎉

As such, we focused our time on helping them get onboarded, talking to them, and figuring out what we should build next.

What we shipped

• 🆕 You can now unassign actions from yourself. No need to be stuck with those pesky tasks - delegate!
• 🆕 We shipped a Security FAQ page, for all your security FAQ needs.
• 💅 If you create an incident from a message, we now ensure that the message you used is unfurled at the start of the incident for all to see.
• 💅 If you change your name in Slack, we reflect it in the product much faster. No identity crisis for you, my friend!
• 🐛 When using funky quote combinations in summary updates, we now strip them correctly for you.
• 🐛 Sometimes when you assigned an action to yourself, we'd update messages in Slack so it looked like you had also created the action. We fixed it so we don't re-write the history books.

Aaaaaand breathe.

We launched last week, and it went quite a bit better than we expected!

We've been inundated with support, feedback, potential customers and ideas for what to focus on next. However, pride comes before a fall, so we're being careful to keep focussed on talking to customers and building.

We spent the week putting together the final polish on the website before we went live. We've also been preparing the product ready to put it in the hands of early access customers, so we've been dotting the i's, crossing the t's and wrangling the experience to make it "just so".

What we shipped

• 🆕 You can now create incidents directly from messages and alerts.
• 🆕 Don't like our #incidents channel and want to name it #hamsters instead? You can now do so.
• 💅 Slack messages are now more nicely presented in the timeline, especially for links and user references.
• 💅 Durations appear more nicely formatted for incidents that stay open over 24 hours.
• 💅 Dashboard buttons don't resize weirdly on smaller screens any more - we hear you, Macbook Air crowd!
• 👷🏽‍♀️ We sped up our build and deployment pipeline. We can deploy code to production automatically in less than 5 minutes from merging a pull request. We hope Charity Majors would be proud.
• 👷🏽‍♀️ We made sure that we'd get paged and a Sentry error raised whenever something went wrong.
• 👷🏽‍♀️ We did a full security review of our application and cloud infrastructure.
• 👷🏽‍♀️ We made sure that we'd get paged and a Sentry error raised whenever something went wrong.
• 🐛 The incident bot no longer shows up as a participant on the incident homepage (we prefer to stay in the background).
• 🐛 The timestamp for the last update to the summary formats the date correctly.
• 🐛 Slack pins show up on the timeline in the order they were sent, rather than the order they were pinned.
• 🐛 The Settings dropdown used to stay open even when politely asked to close. We no longer ask politely.