At 10am this morning (for a period of 2 hours), we forced the Cloud Support team to work remotely and switched off their VPN connection for the duration. We told them they have to provide full seamless cloud support cover as well as performing their duties. You are reading this and probably thinking “What the….? Let me call and find out what you are playing at.”
Before you do, hang on a moment; read on and all will become clear. The Cloud Support team was testing its Business Continuity Plan, covering the scenario where Hornbill HQ became inoperable due to a disaster (e.g. an explosion). We didn’t actually blow up the building — that would have been silly — but we carried out a simulation as accurately as we could. In preparation for the test, we set up a secondary admin network in a data center in Maidenhead, which is available 24 x 7, to allow the team to RDP to customer instances if required. (For security, we lock down server access to a subset of the Cloud team who must be VPNed into our network.) In addition, we set up a new instance of ITSM in our cloud and a backup email address. An external Nagios monitoring server was used to monitor all services to ensure that any problem found would be acted on before it impacted you.
Throughout, our Information Security Manager was in direct contact with the team, both by phone and through Supportworks ITSM. He was monitoring progress with the authority to suspend the test should any customer become affected. Whilst monitoring he was also logging a ridiculous number of support calls to keep the guys on their toes and to make sure that the responses were timely.
Everything went according to plan. The test ran to completion and we gave our customers the same level of support we provide day in, day out, 24 hours a day, 7 days a week, 365 days a year. We have a bunch of observations and suggestions for improvement that have been fed back to our Information Security Manager to log. We are planning to repeat the test again in a couple of weeks’ time as part of our continuous improvement.
A month ago we tested another scenario: dealing with an outage at our data centers. We successfully demonstrated that customer instances could be restored from backup to an alternative data center in a timely manner. This test will also be repeated, as we have made some additional improvements to our application-level backup system.
We are committed to testing the Business Continuity Plan at least twice a year. The Information Security Manager has even written this into our ISO 27001 Information Security Manual. We will keep you posted on our progress, and in due course will ask some of our customers to participate in a future test.
If you have any comments or questions about the Business Continuity tests, please feel free to contact firstname.lastname@example.org