We have to come clean.
During KubeCon, we experienced an incident that we weren’t ready to discuss until now. This incident caused quite a disruption and, had it been left unresolved, would have had a massive snowball effect.
At the time, we didn’t want to raise any alarms, so we kept it quiet while our team rallied to resolve it. And to be honest, most folks probably didn’t even realize that it happened since we moved so quickly.
Really, this was a massive effort made more impressive by the fact that most of our team had never dealt with something like this before. It was truly unprecedented.
If you’re one of the few to actually be affected by this incident, we apologize.
For the rest of you, it’s time to admit what went wrong.
At KubeCon…we ran out of t-shirts. Here’s the story of how that happened and how we rallied to give the people what they wanted: swag.
"...wouldn’t it be funny if we ran out of shirts?"
Going into Day 1 of KubeCon, our team joked about how funny, albeit unlikely, it would be if we ran out of swag. We even tweeted about it, and tested our Status Page with an incident, which was probably a sign of things to come.
Typically, we try to avoid foreshadowing our own incidents, but it happened here–a misstep that we won’t repeat and one to note for our post-mortem meeting.
“Just kidding, we have plenty of shirts. But you know we just had to test our brand new Status Page…”
Anyway, we felt more than adequately prepared for the conference.
We had hundreds of T-shirts in various sizes and couldn’t imagine running out over the course of three days, let alone one.
But then…
The first signs of trouble
Day 1 brought a tsunami of attendees–which was great! What we didn’t anticipate was an overwhelming consensus that our T-shirts were among some of the best at KubeCon.
We were even featured in a tweet listing the top 10 swag at the conference.
Needless to say, we were a crowd favorite. But this recognition was also the start of one of the gravest incidents we’ve dealt with in our history.
We began giving T-shirts away pretty liberally until around mid-morning when we looked around and realized that we were running to our storage room to grab shirts more often than we’d liked.
Put another way, we were running out of shirts. Fast.
#inc-2023-04-19-we-need-to-get-more-t-shirts-to-kubecon
Come 11 AM local time on Wednesday, our worst fears had come true. The situation we joked so much about was waving right at us, ironically so. “...wouldn’t it be funny if we ran out of shirts?” we had said jokingly. Not realizing we were predicting our destiny.
By 11:05 AM, we were swag-less; without swag.
But we still had two more days left at KubeCon. So we did what we do best: declare an incident!
At first, we decided to get a courier over to Amsterdam from London to deliver a massive bundle of shirts. But we soon realized how costly, risky, and inefficient this would be.
Plus security at KubeCon was tight, so there wasn’t even a guarantee that the courier would even be able to get into the venue.
We were running short on time–we needed more shirts in just a few hours. So did we do? We hopped onto Skyscanner and started searching for flights.
One of our team members, Anna, had volunteered to jump on a plane with 2 hours' notice.
She was going to Amsterdam.
"Does anyone have a suitcase?"
Thus began the mad dash to pack as many suitcases as possible with T-shirts, without raising alarms at local customs or having to fill out time-consuming paperwork to declare goods for transport.
One of our engineers, Issac, volunteered his suitcases for packing and we placed an order for 1-hour delivery of packing cubes to maximize space.
Packing had commenced.
Within an hour, our storage locker at incident.io HQ in London had been completely cleaned out. We had no shirts left.
A while later, we were ready to go!
"The swag has departed"
By 1 PM Wednesday afternoon, our T-shirts had been packed and out the door of our London HQ.
The easy part was over—but in many ways the journey had just begun.
Now Anna had to make it through customs without a single hitch and successfully deliver the swag package. We all waited anxiously for updates in our incident Slack channel.
Thankfully, she made it through with only one minor issue. Her suitcase handle broke, but no incident was declared.
We were back in business!
When in Rome Amsterdam
A short flight later, Anna was safely in Amsterdam and on her way to the conference hall. We replenished our swag boxes, and all was well in KubeCon land.
On a serious note, this entire situation was a massive effort by lots of folks. Typically, preparing to ship swag over to a conference is a weeks-long endeavor, so the fact that we rallied to get it done in just a few hours is a remarkable feat of teamwork and grit.
And at a high level, we really think that this highlights the versatility of the incident.io product.
Sure, you can and should declare an incident for serious issues, but the product is designed to be used how you see fit. If that means declaring an incident when you run low on supplies, do so! As long as the issue gets resolved faster, who are we to tell you otherwise?
Hopefully this doesn’t happen to us again, but at least we have the #inc-2023-04-19-we-need-to-get-more-t-shirts-to-kubecon incident channel to reflect back on–and laugh a little, too.
Thanks, team, for not panicking.
