6 Sysadmin Tales From the TrenchesJuly 28, 2017 11:49 am ·
Today is System Administrator Appreciation Day, a day put aside to celebrate and appreciate the tech workers who labor behind the scenes and make sure our day-to-day technology tasks continue to work seamlessly, without interruption. Though we know our resident sysadmin team handles hardware, firewalls, troubleshooting, and the like, we may not realize some of the things they’ve had to tackle in their work — from the ridiculous to the befuddling.
In an effort to shine a light on this hero demographic that keeps our computers up and running, we reached out to some community experts to see what they had to say about their craziest and coolest experiences on the job.
I Spy a….
I had a user complain that their PC just stopped working. Checked the usual suspects like power lead and plug fuse. Opened up the case to find a fried mouse! —Andrew Wright, IT Coordinator
User Error Gets New Meaning
I once had a call from a receptionist of a small accounting firm I looked after, complaining that their internet stopped working. After taking almost five minutes to get her to open a command prompt and type ‘ping google.com’, I went a step further and talked her through doing the following:
- Make sure the modem was on (they kept it and their server under her desk, of all places, where it was easy to accidentally kick out the plugs). Yep, it’s on!
- Reboot the modem by turning it off, waiting 30 seconds, and turning it back on. Nope, that didn’t help!
- Verify which lights on the modem were on with her. Yes, all the lights that should be on are on!
After about 10 more minutes on the phone, she begged to have me visit their office and get the problem fixed because “the boss was getting cranky”. No problem, says I, so I jumped in my car and drove the 20 minutes or so to their suites. Upon arriving at their suites, I headed straight for her desk and jumped underneath to investigate while she trotted off to make me a nice hot cuppa.
Doing my best to ignore the smell of feet and pushing aside all the spare shoes cluttering the space under her desk, what did I find? The power plug for the modem had been kicked out, of course! I plugged it back in, accepted the praise about what an IT genius I was, finished drinking my cuppa and headed back to my own office.
Upon returning, I raised the following invoice: Emergency call out to repair internet connection. Resolved by plugging modem back into power point.
I was expecting the owner to query me on that bill, but it was paid without a single complaint. Go figure.
Yes, this is a true story! —Andrew Leniart, Director of Andrews Computer Help Zone
View From the Other Side
Not sure if this really counts, because I wasn’t the sysadmin in this scenario; instead, I was the problematic user! But here goes: back when I was a student at CalPoly, we had access to the school’s UNIX system for things like our email. I had taken the C and UNIX class, so I knew my way around the command line enough to do more than just use PINE to check my email. I also really liked the SETI @ Home project, so I got the UNIX version of the client and installed it on my account on the school’s system and would just let it run.
I think, maybe, I had it going for a month and I have no idea what sort of resources it was taking up. I imagine it wasn’t really all that bad, but eventually, a sysadmin did shut it down and told me not to do that again…I imagine college sysadmins have to deal with a lot of shenanigans like that!
—Brian Matis, product manager at Experts Exchange
I Swear, It Won’t Work– Wait!
Believe it or not, my ex boss once asked me if I was putting bugs on his machine that I deactivated when he called me over, because I’d get there and ask him to show me the problem, only to watch everything work fine while I watched! True story! —Andrew Leniart, Director of Andrews Computer Help Zone
Knock Knock, Who’s There?
Got a support call, obviously late on a Friday afternoon. ‘Everything stopped working’ they said, and ‘nothing was changed’. The investigation confirmed just that, nothing did work but something was changed. After digging a bit, I found the problem: Access this computer from the network was set to an empty group, meaning nobody could access it. On the plus side, I am glad they were hardening their environment.” –Shaun Vermaak, Technical Specialist / Developer
Time to Step Out of the Box
I want to share a wild duck idea, I executed during an Exchange server migration from Exchange 2003 to 2010.
I was leading an Exchange 2010 migration project for one of my clients. This client had several sites running Exchange 2003 across various countries. There were several sites running with a very low bandwidth internet link. This client had servers on oil rigs with low satellite links, too.
During the Exchange 2010 migration project, when we planned to start Public folder migration from Exchange 2003 to 2010 server, I needed to add a replica for new Exchange 2010 Public folder servers. When we added it, it started replicating across all sites, due to the low bandwidth site, the internet went down completely. I was asked to stop this replication, however, in Exchange 2010 — due to change in product design — we didn’t have the option to stop the replication. Like in Exchange 2003, we had a separate routing connector for Public folder replication. In Exchange 2010, everything is tied with AD site/services. Secondly, my customer didn’t want to stop replication to a high bandwidth site.
To fix this issue, I reached out to Microsoft for help and spent nearly a month [waiting] and finally Microsoft told me they could not do anything with my situation and we needed to live with it. I went to Microsoft’s coding team, but no luck.
Wild Duck idea:
One day, while driving back to home, I was thinking about what I could to do to stop this replication issue to the low bandwidth site. And boom, I had an idea! The next day, I called Microsoft and my client, brought both of them into one call and told them my idea. Microsoft was stunned with my idea and the customer was really happy.
What was the idea and solution?
I suggested that we create two PowerShell scripts:
- Script 1 will be filtering public folder messages pointing to the low bandwidth site and freezing the queues. It will be running from two parallel servers during production hours.
- Script 2 will be releasing the queue after production hours.
I implemented the scripts and tested for one day. After implementing the scripts, I fixed the replication issue to the low bandwidth sites and continued with the remaining project. I considered this one of my best ideas, where even the vendor was unable to provide any solution.
- Project which was on hold for more than a month, finally started and we finished before the target date.
- I got several appreciations from client for my wild duck idea.
- I was awarded Bravo Techno Award for my idea. —Amit Kulshrestha, SME-ECS