Reworking our Terraform provider

We're working through our Terraform provider resource by resource to make your infra-as-code setup easier to understand and maintain.

Alert sources

New resources: incident_alert_source_beta and incident_alert_source_attribute_beta

The main improvements are:

• Attributes are split out from their owning alert source, and each attribute now holds the expression(s) it uses.
• Improved handling of rich text: you can now write templated strings and markdown instead of a full prosemirror AST.
• The alert source config is validated at plan time, surfacing any errors that would cause the apply to fail.
• Some improvements to Terraform syntax sugar, to make it a bit more readable.

The old incident_alert_source resource is still around and working as usual, but we'll deprecate it once we're happy with the new ones.

Schedules

New resources: incident_schedule_beta and incident_schedule_rotation_beta

The big improvement here is removing "rotation versions" from Terraform. Adding or removing someone from a rotation now means editing the lineup in the rotation resource, and optionally choosing when the change takes effect: immediately (the default), after the current shift, or after a full rotation.

We also make it obvious when you’re about to make destructive changes such as when a rotation change would drop previously configured overrides.

We’ve also made it much easier to understand the Terraform plan so you have confidence in the change you’re making, and we now validate the schedule at plan time.

Escalation paths

New resource: incident_escalation_path_beta

Once again we’ve introduced a ‘validate’ step so that the full escalation path is validated at plan time, to reduce unexpected failures on apply.

We’ve also made it easier to read the conditions syntax by adding first-class support for conditions like priority_one_of and working_hours_active.

Tell us what you think

We’ve shipped these resources in beta so that we can continue to make improvements over the next few weeks before we mark them as ‘stable’.

So if you're managing incident.io with Terraform, we'd love to hear about any rough edges you’re still experiencing with the new resources. Talk to your CSM, or come and find us in our community Slack.

We’ll be working on Workflows next!

On-call

Alerts list with more escalation awareness

The alerts list in the dashboard now knows a lot more about related escalations. You can:

• see paged users and attached escalation states directly on the list
• filter alerts by the user they were escalated to, so you can find the ones that paged you
• filter to alerts with unacknowledged escalations

Better Grafana alert screenshots

When a Grafana alert fires we attach a screenshot of the panel that triggered it. We now render through Grafana's dedicated single-panel route, so you get the panel on its own, filling the frame.

This applies everywhere that image shows up: the alert details page, the alert pulse message, the pinned summary in the incident channel and when talking to the agent.

Use a single API key for multiple heartbeat monitors

You can now create a single API key to send ping requests to many heartbeat monitors.

Previously each heartbeat had its own unique alert source token, which could be tricky to manage if a team had lots of different heartbeats.

Sync a Slack user group with the next on-call person

When you sync a schedule to a Slack user group, you could already pick 'Currently on-call' and 'Everyone on the schedule'. Now you can pick 'Next on call' too, so the handle holds whoever is up on the next shift rather than the person currently on it.

Email alerts include any image attachments

Previously, email alerts only preserved the text content of the email. Now they also show you the image(s) included in the email, which can be really useful if it includes a screenshot explaining the error someone’s experiencing.

These images are also visible to all our AI features, including Investigations.

Nexus

Telemetry support for Logz.io

Investigations customers can now connect Logz.io as a telemetry source provider. Customers who use Logz.io can connect their account to give us access to their logs and metrics. Read our docs to get started.

Audit logs for telemetry and connector access

We now generate audit logs for telemetry data source queries and requests, and for calls to the extension connector tools. This builds on the existing audit logs for installing, uninstalling, and changing write grants.

If you wire up telemetry or extensions in Nexus, you can now track who accessed what and when — handy for security teams reviewing what they're connecting.

Response

Incident subscriptions via WhatsApp

Users can now subscribe to incidents and receive updates over WhatsApp, so your stakeholders can follow along wherever they already are.

Support for ServiceNow dropdowns

We now show you the available options in a ServiceNow dropdown (a 'choices' field in SNOW). That makes it much easier to set up your configuration correctly, so we send the right value to ServiceNow rather than you having to dig through your SNOW configuration to find out the choices are actually [1,2,3] under the hood.

You can also sync a ServiceNow field to the catalog directly from the incident ticket template. If a choice list changes in ServiceNow after you sync it, the change comes across automatically.

Create incident channels in different Teams within Microsoft Teams

Microsoft Teams customers previously had to pick a single team where all incident channels live. Now you can use an expression to determine which Team to use.

Note that we can't move channels between teams (as Microsoft’s API doesn’t support it), so this applies to the incident at the point the channel gets created.

2-way sync for Jira and Linear incident tickets

Comments posted on a Jira or Linear incident ticket now show up in the "all activity" view of the incident, and get posted back into the channel.

Caveat: Comments won't sync instantly for customers connecting via our Atlassian Marketplace app. Connections via OAuth will have a sync within 60s of a new comment getting posted.

Post incident updates from mobile

You can now post an incident update straight from the mobile app. Open an incident's updates tab and there are two buttons for writing one.

Previously the only way to share progress from your phone was to type a message into the chat, which was a roundabout way of getting there.

Status Pages

No more TLS errors when you migrate a status page

Previously, we'd only generate a TLS certificate after you migrated your status page to us. That meant anywhere from a couple of minutes up to half an hour of serving scary "invalid certificate" errors on a brand new status page.

Setting up a custom domain now strongly encourages you to complete a DNS-based challenge, so we can get a certificate first and then activate your custom domain.

What else we've shipped