Tales from the Racks: Sysadmin StoriesJuly 29, 2016 8:57 am ·
Sysadmins are all too often the unsung heroes of any given organization, quietly working behind the scenes to ensure that the whole thing doesn’t go up in flames. In honor of all of you on this, the 17th Annual Systems Administrator Day, we wanted to share some tales of valor from within our community. While none are quite as dramatic as this cautionary tale, it’s clear that we would all be lots without them.
Happy Sysadmin Day, everyone!
Joe Frixon’s story:
I would like to share an experience which actually directed me to pursue a successful path in my career. This was the time when virtualization had come to a larger audience with the VMWare GSX hyper-visor and ESX for enterprise. One of my HP DL160 1U server’s SMPS (power supply) had gone bad, and unfortunately that server had only one power supply. The replacement was expected a week later but the incident couldn’t even give a single day of leeway, as this server was hosting our only PDC Domain controller (Windows 2003). With this checkmate I was forced to choose a path of virtualization. Until now I was not a fan of virtualization as I wanted to stick to the old school experience of having machine critical functions on physical servers.
VMWare had this wonderful utility called P2V conversion. But how will i boot if the SMPS is not working? Fortunately we had a similar server running SQL DB which we graciously shut down and did a swap for an hour and converted the server to and ESX box (evaluation version) and made it functional when the converted server started it worked like charm after a couple of reboots. Now this made me to choose virtualization as a reliable alternative to the deployments i have planned for all my future deployments.
Mohammed Hamada‘s story:
I have to tell a story that happened with me not so long ago. I was doing an Office 365 migration project from Exchange 2010. The project consisted of 2 phases. Phase 1 was to migrate Exchange users to Office 365. Phase 2 was cross forest AD migration from Forest A to B, then integrate the forest B with Office 365.
The Hybrid setup went pretty smooth and everything was working fine for the first week and I started migrating users but the customer told me that they recently purchased Exclaimer for central signature solution and that during the Office 365 meet with a Microsoft Presales man. the Presales engineer told them that it’s possible to use Microsoft’s Disclaimer (Mail flow rules) to sign emails (Which was partially true).
I got stuck there for few weeks since they stopped the project as they proceeded talking to MS regarding this and how they’d get reimbursed for the presales engineer’s mistake. They asked me if it’s possible to use the disclaimer as a signature and we did use it with some HTML code but still the disclaimer is not meant to be used as a signature.
Before I finish configuring the disclaimer, One of the IT guys who was responsible for Exchange and Vmware administration called at midnight and he sounded pretty weird. Once I asked how can I help he mentioned that he mistakenly deleted all VMware’s Datastores including those where Exchange DB and Servers were at .
Our company’s VMware team headed there and I with them as well but since their IT admin deleted even the backup datastores .. there was nothing left to restore or recover from .. The only way to save the data was to activate all their Office 365 licenses, Convert their OST to PST files, configure new Outlook profile and import all PST files. which is exactly what happened.
This took about 2 weeks to finish but data loss was almost 0% Thanks to the OST copy.
After that I learned to tell my customers to always keep the cache option enabled all the time and not for one year only as it’s by default in Outlook 2013 and 2016.
Brian B‘s story:
Working for an IT services company, I was called in to look after a client. The client had a Novell 5.5 server that had blown a drive controller. To top it off they were using Notes. Not that there is anything wrong with Notes, but in trying to find out more about it, another Notes expert referred to this version as follows: “Windows ME was for Microsoft what this version of Notes was for IBM”. I can’t remember the whole reasoning, but we couldn’t get their notes back online without getting their Novell environment going again.
First step was setting them up in a new AD environment with outsourced email. That was easy, but I still had to get their old files and Notes back.
At least they had a tape backup of their files. Server was too old to find replacement parts though. So first had to find a machine that was on the compatibility list. Next challenge was the only Novell install media they had was an upgrade, so wouldn’t install by itself. Spent quite a lot of time finding some compatible hardware and Novell install media.
The biggest frustration was listening to the other office employees talking to each other and even telling their customers about how they couldn’t do anything, or had to do things manually, despite me taking the time to update most of their processes to the new systems.
All the time the nervous owner of the company was watching how things were going. In the end we got everything running again, but just barely. Long enough to get files restored and keep the notes database up long enough that things could be transferred out. Then they sold the building and moved, but that’s another story.
Thank you to everyone who shared their tales from the racks. If you have any stories to add, email us at firstname.lastname@example.org!