On top of the usual collection product improvements, this week we've also shipped our new blog. The old one felt a little uninspiring, and since we've got lots we want to share, we thought a little overhaul was a good investment in time. To mark the occasion we shared the first in our interview series, this time featuring Colm Doyle from Slack.
We've had a few things competing for our time over the last few weeks, which has slowed the pace of change a little. Next week's looking pretty clear though, with plenty of time to devote to product and engineering. Stay tuned!
/incident update, we now prompt you to check that the overall summary still makes sense. A well written summary can go a long way to keep folks on the same page.
Over the last week we've been busy making it even easier for your whole organisation to onboard with incident.io. Whether it's simple step-by-step instructions for the installer, or helpful messages to folks after they've participated in their first incident, we've got you covered.
/inc update, we'll now prompt you to check the incident summary still makes sense. A few minutes getting everyone on the same page can save hours later on.
/inc updatecommand whilst also changing the severity, we'd show the old severity in the closure modal. Now, thanks to a report from the folks at Ziglu, we show the correct one.
Instead of doing just one thing last week, we made lots of smaller improvements to the product. We also got a lot of internal admin off our plate to keep the paperwork gods happy and keep the wheels turning.
We won't bore you with the paperwork, but we hope you enjoy the new features and improvements!
/incident statuswill still work just fine.
Incidents are a team sport, and to help everyone get involved we built incident actions. With a quick
/inc action in Slack, anyone can view, create, and pick up actions, making it easy for folks to self organise around whatever needs to be done to get things back on track.
We got some feedback that the actions interface in Slack could be a little clearer though, for example the layout was a little cluttered and it wasn't easy to see who created what. We've now refined the design and tweaked what we display so it's even easier to get a clear picture of what's going on.
Got thoughts on this or any other part of incident.io? Jump into our community Slack and let us know!
/incident helpfrom outside an incident channel, we used to open the create incident modal with an incident title of "help". You were probably after some actual help, so we now show you modal with some tips. Thanks 100/x for this feedback!
We're fanatical about great support here about incident.io. Every customer gets a shared Slack channel to make it easy to ask questions and give feedback. We've now made that even easier by adding Intercom to incident.io.
Whether it's a question about how to add a custom incident role, or feedback on an improvement we could make, you can now send that to us in a couple of clicks. Click the button in the bottom right hand corner of incident.io to get started.
Behind the curtain, Chris, Pete and Stephen will be diligently answering your questions, fielding your feedback and blushing at your compliments. Looking forward to hearing from you!
Good incident response is about good communication. In incidents, it's particularly helpful to provide a small, structured update to the organisation on a regular basis. This keeps everyone on the same page, and helps resolve worries about when the next update will be coming, too.
We're always looking for ways to make good communication easier, and a few customers have mentioned that using seperate commands to udpate the severity, status and summary often felt a bit harder than it needed to be. You might not remember to update the status, or forget to update the summary after new information came to light — we certainly did!
We think all these fields work better together, so last week we set to work on incident updates!.
Updates are an easy way to provide a small, structured update to the rest of the organisation on a regular basis. We tie together existing fields like Status and Severity, with a new field allowing you to share any other context. Additionally, if you set a reminder for when you're expecting to send the next one, we'll remind you at just the right time ⏰
Hopefully this makes internal communications a lot easier, and reduces some of the burden when it comes to communicating regularly. Give updates a spin in your next incident, and let us know what you think!
It's a been a while since we last posted one of these, and despite the radio silence we've been busy behind the scenes at incident.io HQ. Firstly, we now actually have a HQ, and it's in an old firestation - very fitting. Pete and Stephen have left their full-time jobs and are now working 100% on incident.io.
This week we've decamped to an eco-lodge in the Wye Valley to plan for the next few months. We've got hundreds of ideas and a mountain of valuable feedback, so we're taking a week to refine it all into something amazing.
No more breaks now. Expect weekly changelogs to be weekly. All systems go.
We received some lovely feedback from customers over the past week — this kind of thing makes our day, thanks folks!
We at @Farewill started working with the fantastic @incident_io team earlier this year, and honestly it makes life SO much easier to have seamless tooling when everything is on fire. Watching our engineers working in the incident channel to check how we're affected is just 🤩— Sally Lait (@sallylait) June 8, 2021
Check out @incident_io to manage incidents within slack. Awesome workflow and all the automation you ever wanted to bring more structure and fun into a potentially stressful situation. We'll start using it for every alert @giantswarm now.— Timo Derstappen (@teemow) June 12, 2021
Super awesome demo of @incident_io from @sjwhitworth earlier. Really excited about what the team is doing to switch up incident management by bringing together tools we use & love into a streamlined workflow🔥🔥— Nick Coates (@nickcoatesuk) June 10, 2021
We now have a number of customers using us daily to respond to incidents, and we're seeing more of you leaning into improving your post-incident followup. This is an area where until now we've been heavily opinionated, without much room for customisation - something we always knew we'd need to change.
We want to make it easy to imprint your process directly into incident.io - we want you and your team to feel like this is your tool, not someone else's.
We've always been able to automatically generate incident documents (also often referred to as post-mortems) and timelines automatically, but now you can now customise the format to your heart's content.
closed_attimestamp that we use for tracking. That was silly. It now doesn't do that. Thanks for helping us spot this one, Farewill!
In incident.io, roles are a first class citizen. We think of them as 'hats that you can wear' during an incident. We use them to make it clear:
Until now, we only supported a single role - the Incident Lead. We knew this was limiting, and our customers agreed, so we fixed it.
You can now add and customise your own roles to fit how you like to respond to incidents, attaching a description and instructions on how to play the role. Roles can be assigned and re-assigned during incidents by using
When someone is assigned to a role, we'll privately send instructions to the assignee to make it clear what they need to do, and help them get up to speed.
/zoomin an incident channel, we'll prompt you to add it as the call for the incident. Like magic.
@incidentin our codebase.
When we started incident.io, we felt passionately that we should replicate the feel of human communication during an incident. Need to escalate to someone? Just talk to us with
@incident escalate and we'll guide you through it. The drawback? We made a lot of noise in your incident channel 🙉
We've listened to your feedback, and today we're making the move to Slack's slash commands. Whilst it's just a single character, moving from '@' to '/' has a big impact. When you
/incident escalate it makes no noise in a channel, and as an added bonus we can take you straight to the right place. It's a win-win!
As if that's not enough, we're also introducing the Incident Home. Can't remember the command for the status page? Just type
/inc if you're in a hurry) and we'll help you get to the right place.
Most of the energy around incidents is, quite rightly, focussed on coordination and solving the problem, but there's a tonne of value in what happens afterwards, too.
Whether it's to spot topics for a postmortem, or learn which parts of your incident response could be improved, just asking "how did we do" is a powerful tool.
Often we forget to ask, and it involves a lot of additional admin to follow up and collect responses, but from today, incident.io will handle it for you.
We'll automatically check in at the end of an incident and ask everyone how it went, with a few specific questions, as well as some more general prompts.
This is just a first pass, but we've already seen customers using it to gather ideas for improving their incident response. Let us know what you think!
Our main focus this week has been making sure it's easy to learn what incident.io is, and how it works. We've spent a lot of time working on great tooling for incident leads. However, by definition, there's only a single incident lead but many participants and observers. We want to make sure you have a great experience too!
As part of that, we've polished the messages that you get when joining an incident channel to make them clearer and more concise. Additionally, you can indicate whether you're there to help (we'll announce it in the channel if you are), there to observe, or if you want a quick introduction to incident.io.
We'll spend the rest of this week focusing on improvements to onboarding: look out for some new features coming out over the next few days!
call. Give it a try with
/incident lead me. Only a few left to go now..
/incident action, we'd send two separate updates into the channel. We've combined that into one update. Thanks for the feedback, Upvest!
/incident summary Hello world, we used to discard
Hello world. That now forms part of the update that appears in the modal. Sorry about that.
Our customers want a way to train people on incident.io, without unnecessarily worrying others in the organisation that things are going sideways or messing up their lovely incident library and statistics. Until now, you had to use normal incidents to do this. That's no good.
Well, enter test incidents! Test incidents are exactly the same as normal incidents, except:
We think that they're best used for:
Kick off an incident by
/incident test and let us know what you think!
First impressions are important. We've seen more and more people using incident.io recently, which has been fun to see, so we've used this influx of new faces as a prompt to think about what we could do better here.
We're currently focussing on onboarding and adoption to make it easier for new users of incident.io to get up to speed, whether that's when you install us, or the first time you're pulled into an incident.
We'll have more to show next week, but we've already made a start, so you'll probably notice a few changes. If you spot them, let us know what you think!
/incidentto kick off an incident? Too many characters and you're falling asleep by the end? Well, you can now type
/incto save you time, and effort.
/incident recapfor a quick spin!
/incident The website is down, we used to ignore
The website is downand make you type it again. We now automatically populate that into the modal, to save energy for your precious little fingers. Thanks Ravelin, Bud and many others for this feedback!
In 2021, most companies have a status page set up to make it easier to publicly communicate the real-time status of their service to their customers.
However, when you're in the heat of an incident, it can be challenging to remember to update it. Where did we put the login, who has the right permissions, where do I go to update it? These are things we've been asked about time and time again by customers.
To help make things easier, we've brought one of the most popular status page providers out there - statuspage.io - right into your incident channels. Type
/incident statuspage to notify customers of new incidents, provide updates to ongoing ones, and to let them know when the impact is over.
To get started using this for your incidents, head over to our shiny new documentation which takes you through how to install and use it.
Statuspage has a great feature inbuilt that allows you to pre-fill the impact, update text, and affected components. This makes it less stressful when figuring out what to write. We don't currently support this, but we hope to have it out in a couple of weeks. If this is something you'd like to see, let us know!.
We've been on a bit of a roll recently when it comes to adding integrations. Over the past month, we've added Opsgenie, JIRA, GitHub and Linear. Well now, Clubhouse gets to join the club!
You can use our Clubhouse integration to export actions from your incidents, to aid prioritisation, and make follow-up a breeze. When you've exported your actions, we'll continue to track and sync actions into incident.io, and link back to the incident from any issues we create, in case you need to reference it later.
One of the things we knew wouldn't last long was "open/closed" modelling of incident states.
That's not how any incident we've ever been part of has worked and although it got us off the ground, we've been embarassed about it ever since.
Your feedback told us the same thing, and we decided it was high time we fixed it.
Incidents can now be in several states:
These states can be controlled via Slack, and the timestamps can be updated via the web UI after the incident if any adjustments are needed later.
We think they're really useful for clearly communicating the progress of an incident to your team and stakeholders. Additionally, capturing this data lays the foundation for lots of interesting and useful insights, and we look forward to sharing those with you in the near future.
In the early days of building incident.io, we had a hunch that "storing a document" against an incident might be a useful thing, whether for post mortems, or for something else.
We've since had two bits of feedback repeatedly:
So, that's what we've done. The generic "document" link is no more, and in its place we now support a post mortem document explicitly. Great feedback, keep it coming!
Another mid-release update from us today. You can now export incident actions straight into GitHub Issues!
GitHub Issues provides a developer-friendly, lightweight way to track actions so it's no surprise it's been fairly high on the list of requests from customers.
Once exported, we'll continue to track and sync actions into incident.io, just like you can with our Jira and Linear integrations. We'll also link back to the incident from any issues we create, in case you need to reference it later. Happy GitHubbing!
We were a bit quieter last week because we've been heads down building and responding to feedback, so we're giving you a bumper update this week instead!
Hot on the heels of our PagerDuty integration, we got a lot of feedback from teams using Opsgenie, who felt a little left out.
We're delighted to say those days are over, and you can now escalate to your team via Opsgenie from right within your incident channels. Just like with PagerDuty, let us know who you need, and we'll handle the rest.
We've also got a few other nice integrations and improvements we'll be talking about more later this week.
A little mid-release update from us today. You can now export incident actions straight into Linear!
We're big fans, and selfishly we're pretty excited we can use this ourselves, as copying things over manually after incidents was getting a little tedious...
Once exported, we'll continue to track and sync actions into incident.io, and shortly we'll show you insights into how many actions you create, how many get followed up, and how quickly, too.
We'll also link back from tickets to the incident, in case you need to reference it later.
We want to meet you where you work, which means gluing together the tools you already use. We've got a long list of integrations planned, and Linear was getting a lot of demand, so we bumped it up the list.
Getting this kind of early access feedback to help us steer the product is really what drives us and makes building incident.io worth it, so thank you! If you have more integrations you'd like to see, drop us an email or let us know on twitter.
We bring you this update under the rays of some glorious spring sunshine 😎🌼.
One of the problems we hear time and time again is that following up on actions after an incident is harder than it should be. They get written down in slack or a post mortem document, maybe exported to an issue tracker, but more often than not, are then left to languish.
Without a clear link back to the incident that caused them, it's hard to track whether or not they get done, let alone see how long that takes and be able to report on it over time.
As a first step towards solving this, from today you can manage your actions via the incident homepage, and export them directly to Jira. We link them back to the incident that they were created from, and keep the state in sync for you.
Next up, we'll be building tools to help you make sure they get done, and surface insights so you can truly understand how quickly that's happening.
We've started with Jira, but Linear and others may follow soon. If you're keen to see that happen, let us know!
/incident help, help will be returned in a modal that only you can see.
"Resolving our incidents is only half the story"
That's what you keep telling us, and we couldn't agree more.
To really get the value from your incidents, you need to follow up effectively. Right now, we're focussed on building out the tools to help you do just that. We're starting with extracting the insights and learnings which, let's face it, you've already paid for. We want to help you get your money's worth.
A common solution we've seen (and used ourselves!) is what we at incident.io HQ refer to as "The Google Sheet of Doom". Often this is a manually compiled spreadsheet, containing some high level numbers on things like duration, severity and impact, maybe with some graphs and Pivot Tables thrown in at a stretch.
We think we can do it better, and do it for you. Today we're taking the first step in that direction by launching the first version of Insights, available for all incident.io early access customers.
We love our bot. It takes rote work off our hands, and let us focus on the things humans are great at. However, it's important that the incident channel remains clean and easy to scan, and it were getting a little loud for our tastes.
We started the week deep in thought, coming up with design principles for how the bot should interact. We'll be writing about these soon - stay tuned to our blog for more.
Armed with these principles, we built them into the product. The bot should be noticeably quieter. Let us know what you think!
Do you ever find yourself rummaging through an incident channel for a dashboard someone shared, a runbook the team referred to earlier, or that documentation you were reading a few minutes ago?
We do — all the time.
From today, you can just type
/incident links and we'll scurry off and find them all for you. Let us know what you think!
It's often the case that you reach the boundaries of your knowledge during an incident, and you need that special someone to help you out. We've all been there. However, getting help can be hit and miss: tagging people in Slack, or trying in vain to blow the dust off your PagerDuty login.
Fear no more - you can now page users and rotations directly in Slack via /incident escalate. We'll stream notifications and acknowledgements into the channel as they happen so you know who's on their way.
We've launched with PagerDuty to get ourselves comfortable with the flow, but we'll be launching Opsgenie and VictorOps soon!
We've been getting tonnes of great feedback from our early access customers this week. We also said hello to 4 new ones. 👋🏼
Incidents can be fairly stressful, so the fact so many of you not only fixed your own bugs, but also helped us find and fix a few of ours is something we're incredibly grateful for.
We've been spending a bit of time thinking about how we can best share some of the longer term thinking we've been doing on what's coming next. This week we're really excited to launch our incident.io product roadmap, which lets you get a view into what we're up to.
There are upsides and downsides to sharing a product roadmap publicly, but we believe pretty strongly that:
We'll keep on shipping plenty of changes, improvements and fixes every week, but we'll use this as a way to keep you updated on some of the "big stuff" as it takes shape — enjoy!
This week was a particularly special week, as we onboarded our first 4 customers. 🎉
As such, we focused our time on helping them get onboarded, talking to them, and figuring out what we should build next.
We launched last week, and it went quite a bit better than we expected!
We've been inundated with support, feedback, potential customers and ideas for what to focus on next. However, pride comes before a fall, so we're being careful to keep focussed on talking to customers and building.
We spent the week putting together the final polish on the website before we went live. We've also been preparing the product ready to put it in the hands of early access customers, so we've been dotting the i's, crossing the t's and wrangling the experience to make it "just so".