Impact review: Scribe under the microscope

👋 Introduction

In December 2024 we launched Scribe to help responders never miss a detail from their incident calls. By automatically transcribing calls and highlighting key information, Scribe eliminates manual note-taking, reduces time spent getting up to speed, and preserves valuable context for post-incident analysis.

The feature quickly gained popularity among our customers, but with success came an influx of requests for bug fixes, extra functionality, and wider call platform support. Before investing more engineering time, we asked ourselves:

1. Is improving Scribe the most valuable use of our time?
2. If so, which potential improvements should we prioritise?

To answer these questions, we conducted a comprehensive impact review of Scribe, analysing usage data, adoption patterns, and customer feedback. In this post, we'll share our process, key insights, and what they mean for Scribe's future.

🥾The journey so far

Scribe's initial launch supported Zoom and Google Meet, automatically joining incident calls, transcribing discussions in real-time, and surfacing key details via live moments, topics, and structured summaries.

Customers who were up and running with the feature were overwhelmingly delighted with it. As one engineering manager at GitLab put it:

Honestly, it was the thing that made the rollout the best because we had engineers who were just like 'I will never go back. You can't take it away from me.'

However, some customers faced challenges with permissions and configurations, while others couldn't use Scribe at all due to using unsupported platforms like Microsoft Teams or Slack huddles.

Despite these hurdles, the numbers spoke for themselves. In a matter of months, Scribe had transcribed over 5,600 hours of calls across roughly 4,000 incidents, and that number continues to grow. It was clear that we had tapped into a real need, but to fully realise Scribe's potential, we needed to dig deeper.

🤿 Diving into the data

To prioritise the right improvements for Scribe, we needed to look beyond individual feature requests and bug reports. While customer feedback is invaluable, it’s important we are getting a full picture of usage and issues ourselves to truly understand our customers pain points. To get a more comprehensive picture, we decided to dive into the usage data and analyse adoption patterns, user behaviour, and feature performance.

Adoption Patterns

We started by examining the organisation-level adoption funnel to understand how Scribe was being adopted across our customer base. We answered questions like:

How many of our customers use supported call platforms, unsupported call platforms, or no call integration at all?

We found that for customers with supported call platforms, adoption was already quite high at 44%. However we had more customers on supported call platforms that weren’t using it yet, than customers who couldn’t use it because we didn’t support their platform.

Have some customers tried the feature out then turned it off again? If so, why?

We found that retention was overall very high, and we could identify reasons for all customers who had disabled the feature - including permission issues with call providers, and transcriptions not working in their preferred languages.

Are some customers uncomfortable or unable to use AI?

The majority of organisations who actively use calls, but not Scribe, are still using AI features such as suggested summaries or follow-ups. This isn’t surprising, given that Scribe is opt-in but these other features are enabled by default.

User Behaviour

For customers with Scribe enabled on their accounts, we looked at how users were interacting with Scribe. We found that high-severity incidents were much more likely to have an associated call compared to minor incidents, and those calls were also more likely to be transcribed by Scribe.

This suggested that users saw more value in Scribe for critical situations where documentation and shared context were crucial. However, we also noticed that even when Scribe was enabled, it was only successfully joining and transcribing about half of the calls it was supposed to. This pointed to potential issues with permissions connecting to call providers, configurations, or user trust that we needed to investigate further.

Feature Performance

Next, we assessed how well Scribe's key features were performing. We found that users were spending a significant amount of time on calls transcribed by Scribe, with longer durations for more severe incidents. We also saw that the vast majority of incidents with Scribe activity still had active communication in the incident channel, indicating that Scribe was complementing, not replacing, existing incident response practices.

One area where we saw room for improvement was in the ease of use and discoverability of features like transcript editing and call-specific controls. The data showed that these features were rarely used, despite being frequently requested by customers.

Takeaways

Our deep dive into the data confirmed that Scribe was delivering value to customers, particularly for high-stakes incidents. However, it also highlighted several areas where we could improve adoption and feature usage, such as:

• Streamlining permissions and configuration setup
• Providing clearer in-app guidance and error messages
• Making key features like transcript editing more discoverable
• Expanding support to additional call platforms

With these insights in hand, we were ready to prioritise our next steps for Scribe. In the following section, we'll share some of the key initiatives we planned based on this impact review.

🎓 Lessons learned

Conducting this impact review of Scribe was an eye-opening experience. It not only confirmed our belief in the value of the feature but also shed light on areas where we could do better. Here are some of the key lessons we learned:

Clear Guidance and Error Surfacing

One of the most significant barriers to adoption we identified was the complexity of setting up permissions and configurations with call providers. Users often struggled to understand what settings they needed to change or how to troubleshoot issues when Scribe failed to join a call.

To address this, we realised we needed to invest in clearer, more accessible documentation and in-app guidance. This includes:

• Step-by-step guides for configuring each supported call platform
• Prominent links to relevant help articles
• More descriptive error messages that help users diagnose and resolve issues independently

By empowering users to set up and use Scribe without needing to contact support, we can reduce friction for new adopters, and increase trust in our product.

User Control and Flexibility

Another key takeaway was the importance of giving users more control over when and how Scribe is used. While the default behaviour of joining every incident call works well for many customers, others wanted the ability to be more selective.

Some specific requests we heard included:

• Buttons to easily add or remove Scribe from a call on the fly
• Options to limit Scribe to certain incident severities or types
• Ability to pause transcription during sensitive discussions
• Editing and removing transcripts after the fact

By giving users the flexibility to adapt Scribe to their unique needs and workflows, we can drive higher adoption and satisfaction across a wider range of use cases.

In the next section, we'll outline our plans for turning these insights into action and driving Scribe's future growth and success.

🛣️The road ahead

Armed with the insights from our impact review, we got a clear picture of what our users need and how we can make Scribe even better. In fact, we've already shipped some key updates to address the most pressing pain points:

• Manual controls for adding and removing Scribe from calls, giving users more flexibility
• The ability to delete transcripts, ensuring users have control over their data
• Granular error messaging to help users troubleshoot and resolve issues independently

Of course we always have more great features we’d love to build than time to build them, so not every feature mentioned in this review made the cut. Some things we’d still like to build (one day) include:

• Expanding support for more call platforms to make Scribe accessible to even more teams
• Implementing granular permissions and controls, like limiting Scribe to specific incident types or severities
• Enhancing the transcript editing experience with features like granularly correcting or deleting transcript entries
• Supporting transcription in multiple languages

As we continue to evolve our Response product, we'll use the data we've gathered to support our product intuition and customer feedback, to ensure we're building the right things. Regular impact reviews like this one are an essential part of our process, helping us stay attuned to what our users really need.