Tech Horror Stories: It Came From Inside the OfficeOctober 22, 2015 10:59 am ·
It’s just after lunch on a Monday and things are going quite smoothly in your small office. Several new features are scheduled to push live this afternoon and with your manager away on vacation, you’ve been tasked with overseeing the rollout.
You’re going through the usual pre-release sanity checks when suddenly, beyond your monitor you see the entire office rise from their seats and look directly at you. There’s a moment of dreadful silence before they begin moving your way while announcing over one another what the problem is. Two words stand out — each harmless on their own, but dreadful when paired — “Network…down.”
Immediately, you snatch the phone and dial your ISP to ask if they’re aware of any downtimes. “Please hold for a moment while I check.” the technician tells you.
The C-level gang has gathered behind you and are watching on curiously. “Most likely congestion or a damaged line.” you assure them while the strangely relevant hold music blares in your ear.
The technician picks up. “Hello? Looks like everything is up and running on our end.” You nearly drop the phone upon hearing the news and feel your face drain of blood.
“What is it?” asks the Chief of Breathing-down-your-neck Operations.
“The network issue,” you reply, “is coming from inside the office.”
You get to work running through the emergency procedure checklist, attempting to identify the problem. You examine all the usual suspects for these types of outages and are not having any luck.
Since the person who knows the network best is probably overlooking some lush Tuscan valley and sipping on a fine vintage at this very moment, it’s up to you to figure this one out. You recall the words of encouragement your manager gave you on Friday when leaving the network in your care: “Barring anything catastrophic, this will be a breeze for you.” he said, “And just remember, if anything does go wrong, we’ve got our Experts Exchange membership.”
Bingo! Quickly, you pull out your phone, open the Experts Exchange app, and provide a description of what you’re experiencing to the waiting experts. You continue working toward a solution while the experts diagnose your network configuration in search of possible points of failure. If you can locate and fix the problem soon they may not send anybody home and the release could still go out today.
A few minutes pass by and you’re still trying to pin down the issue. The C-levelers, well aware of how pricey downtime can be have started sending people home, and just as they’re gathering their things to head out you notice your question on Experts Exchange has received replies. The first one posted is from a Senior Data Modeler in Belgium who confidently and succinctly diagnoses the issue. You jot a couple things down on a post-it note and dive into the server room to apply the fix. The second it becomes clear that this solution is going to work you sprint out into the parking lot — just in time to stop the first car from leaving.
“Sorry guys!” you shout heroically, standing in front of the procession of cars, “the network is up and running!” And they thought they could just roll out of here…
Oh they’ll be rolling out alright, and with Experts Exchange as a safety net, it’s bound to be a smooth release.