Junior Achievement of New York teaches students about money management and how business works, but the organization itself learned important lessons in both when it experienced catastrophic system failure--despite all the the right preparations. Fortunately, a robust backup, disaster recovery solution saved the day.
The home page of Junior Achievement of New York (JANY) states “[JA] Teaches Young People About Money Management and How Business Works.” Well, JANY nearly learned the ultimate lesson in both when the organization experienced catastrophic system failure. And, what’s worse? JANY and its managed service provider (MSP) made all the right preparations.
Connect Computer of Fairfield, Conn., manages JANY’s systems. When Connect started working with JANY, the first order of business for Lynn Souza, president of Connect, and her team was to evaluate JANY’s existing backup, recovery and continuity solution. It was tape. Given the limited and unreliable nature of tape, Connect proposed a more robust backup, disaster recovery (BDR) solution to JANY--one that could also deliver business continuity and mitigate downtime should a disaster occur.
Connect recommended Datto SIRIS. Based on a hybrid cloud model, SIRIS combines image-based backup, instant local and off-site virtualization, Screenshot Backup Verification™ and Inverse Chain Technology™— all of which yield a superior recovery time objective (RTO). When this was presented to Christopher Malin, vice president of finance and administration with JANY, he was immediately on board to implement the new SIRIS solution, noting that the cost of the SIRIS solution is “very reasonable” and “fits in the NFP [not-for-profit]” budget.
And, yes, even a not-for-profit needs to be aware of its RTO. Consider the fact that Junior Achievement of New York has served a total of 67,850 students and raised a gross total of more than $3.5 million. That’s a lot of students to help, donations to implement and data to manage. Should JANY experience data loss or downtime, proprietary student information would be compromised and future donations would be in jeopardy.
Disaster Strikes, Despite Preparations
When his building’s operations team alerted Chris at JANY that power would be cut off to the building so that the electrical could be upgraded, he, along with Jeremy Truman, Connect senior systems engineer, had plenty of time to prepare and did all the right things. Jeremy remotely shut down all of JANY’s servers. Chris went one step further and actually unplugged everything. This was on a Friday. What could possibly go wrong?
On Monday, Jeremy was on-site with Chris when the power was restored, to make sure both servers, with seven virtual machines, rebooted correctly. Things didn’t go according to plan. One of the two physical servers started. The other failed miserably: Four of the eight hard drives “fell out” of the array, which affected all their data (being a RAID 5). Turns out the drive configurations were corrupt, and Connect could not put the RAID arrays back together without problems and potential data loss.
Jeremy’s and Connect’s first priority was to get JANY operational. With this, they spun up the domain controller, Exchange and Razor’s Edge (a database program) on the Datto SIRIS, locally. Less than two hours later they had three VM servers up and running locally off the Datto SIRIS, and JANY systems were up and running.
Jeremy noted that if JANY had still been using tape, “it would have been days, and then no guarantee the data was valid.” With SIRIS, the team had run a DR (disaster recovery) test a few months prior to this server outage and knew best practices to most efficiently manage the process of recovery. The Connect team stresses that no matter the solution you have in place, “it’s always important to perform DR tests.”
While JANY’s systems were running off the SIRIS device, the technicians had the time and opportunity to rebuild the actual server. This process took 24 hours to complete due to the size of the file exports. With the server repaired, all systems were exported back to the EXSI server and started up, and everything was back to normal.
“No employee knew anything had happened,” explained Chris. JANY did not miss a beat of business continuity. All things considered, a huge success: Datto SIRIS kept upward of 40 users working, student activities running and donations coming in.
Outcome and ROI
The Datto SIRIS more than paid for itself from what started out as a simple server reboot that led to a significant crash and outage. The cost to rebuild the potentially lost data would have taken at least five days of work, not to mention the damaging headlines and loss of confidence in a very respected charity.
Have you ever “crossed the t’s and dotted the i’s,” only to have some unforeseen disaster happen and affect your client’s data? Hope you were prepared.
Holly Wainwright is Director of Marketing at Datto. Datto Inc. is an award-winning vendor of backup, data recovery (BDR) and intelligent business continuity (IBC) solutions, providing technology and support to more than 5,000 channel Partners throughout North America and Europe. Datto’s hybrid-cloud BDR/IBC technology provides instant on- and off-site virtualization of servers and workstations, serving the needs of small to medium-sized businesses.