After extensive planning, the SysAdmin team at Experts Exchange successfully completed the data center relocation to run Experts-Exchange.com from servers based in Virginia. We haven’t moved the employees and corporate offices from our warm, breezy college town of San Luis Obispo, CA, nor are there plans to do so. In Part One of his blog posts on the data center relocation, Site Director Andy Alsup talks about why the relocation was necessary and the buildout process.
The SysAdmins here at Experts Exchange have always believed and practiced redundancy in our configurations with a standby database instance, bandwidth provisioning from both a San Francisco feed and a Los Angeles feed, as well as redundant routers, switches, load balancers, and all the other elements of our software and hardware stack. We did see the need for a backup plan for our primary datacenter located in San Luis Obispo, so in 2009 we identified a co-location datacenter in Ashburn, VA and built out servers and infrastructure that would allow us to sync our critical data to the “other” coast of the United States.
Over time, we refined those processes to make sure we had not only production data, but also mission critical developer services like our SVN repository, build server, and our corporate mail pushed over to Virginia. After a couple of years of refining and validating our processes to make sure that our data was being securely backed up and stored at our secondary data center, we asked ourselves the question that many ask of a DR datacenter – could we do the ultimate test of our DR and actually run our website and make the secondary datacenter our (now) primary datacenter?
Our QA team spent quite some time testing and re-testing our Virginia configuration, which had several new networking elements built into it. Over the last 9 years, we’ve built out our SLO datacenter with some new services, but not much had fundamentally changed with our SLO datacenter, so this new Virginia configuration needed full functional and load testing. After some additional buildout of low power servers, some network redesign, validation of remote management ports and protocols and adding a secondary bandwidth provider, two of us set out on Tuesday April 23rd for Virginia. Once we were in Virginia, we replaced a few failed hard drives, added in some RAM to our webservers, and replaced some failed RAM in our database server. On Thursday April 25th, we turned our DNS Time-to-Live (TTL) down to a much shorter time period in preparation for our DNS/IP change. Now we were ready to work with the team in San Luis Obispo to reset our DNS to point to Virginia. On the day of our long-planned switchover, we were temporarily sidetracked by the sighting of PlayStation Live racks in another cage in our datacenter, but at 1pm PDT Friday, April 26th, we started our downtime in SLO to transition service to our Virginia-based datacenter.
Check back next week on the Experts Exchange Company Blog for Part Two of Andy’s update on the data center relocation. He’ll cover the transition back to San Luis Obispo, where we are now and the lessons we’ve learned.