Hornbill Trust
Call Us: +44(0)208 582 8282

Social: Facebook, Google+

Aug 21 2014

Slow Performance – All UK Instances 21/08/2014 @ 09:10

Due to an unexpected failure of a fibre link to our data centre that hosts a number of UK instances, we are currently experiencing slower than expected performance. The root cause has now been identified as an issue with 1 of the fibre links to our data centre being down, our data centre networks team are currently reviewing traffic flows over the remaining links and migrating traffic to the most appropriate paths to maintain network stability, during this time you may experience some latency whist we move traffic flows around the network.

Update 10:00
Analysis has confirmed 1 effected core fibre is causing significant issues and all traffic is being rerouted to avoid the offending area. At this time no ETA for resolve and we are reviewing our emergency plan (Decision will be made in next hour as to whether we fall over to DR site).

Update 11:00
The issue is still being resolved (Ping times are slowly being reduced as the offending connection is circumvented).
We have reviewed our emergency plan and at this time believe that the inconvenience of moving instances to our secondary data centre would be worse than the slow performance (Would require at least 1 hour of Down time, possible changes to customers firewalls to allow connection to the new hosts and until the DNS propagated analysts\customers would need to use IP rather than Names).

We will continue to monitor this and update you on an hourly basis until resolved.

Update 12:00
The root cause has now been confirmed as a cable cut during road works on the A272 and engineers have been dispatched to resolve the issue. All traffic is therefore being redirected via secondary cables, which is causing load issues (This is also under investigation as secondary cables should be capable of providing full service).

We still believe that falling over to our secondary data centre is not required, however we will perform this on request (please contact Cloud Support Team to request migration).

We will continue to monitor this and update you on an hourly basis until resolved.

Update 13:00
Traffic is still being redirected, however over the last hour Ping times have dropped significantly (and therefore performance increased). We expect this to continue over the next hour and depending on route taken customers may already see the issue to be resolved.

We will continue to monitor this and update you on an hourly basis until resolved.

Update 14:00
Traffic for the most part is now redirected with Ping times < 10ms once again. We therefore believe that continued mitigation of the issue by routing will be enough to ensure that we do not need to switch to our secondary data centre.

Engineers have arrived on site at the cut fibre and are working to full restore communications.

We will continue to monitor this and update you on an hourly basis until resolved.

Update 15:00
All monitoring now shows that traffic\performance has been restored to pre-issue levels resolved by rerouting traffic. These changes will remain in place until the fibre is repaired. We do not expect any further issues with performance, however we will continue to monitor closely.

We will provide a full report to all customers within 24 hours.

Final Update 07:00 22/08
Our data centre team inform us that the effected cable was repaired (2KM of replacement fibre added) at 05:45 this morning and all traffic is following the original routes.



Uptime SLA


Uptime Target


Platform Uptime Last Month
Platform Uptime This Month