Hornbill Trust
Call Us: +44(0)208 582 8282

Social: Facebook, Google+

Oct 16 2017

Possibly Intermittent Issues – Effecting Virgin Media Users – Fixed

During the last 30 minutes we have started to receive calls from our userbase that they are unable to login to their instance.

We have identified the problem as a large routing issue to the data center which holds your instance.

We are working with our data center to resolve the issue and will provide updates every 30 minutes until resolve.

In the meantime should you require any further information please feel free to contact cloudsupport@hornbill.com

 

==Update 14:08==

 

We have now resolved the issue with connections to our data center and this appears to be a routing issue with 1 ISP.

We will be conducting a full review and post mortem to identify any steps we could have taken to prevent the issue.

Jun 16 2017

Issue with performance on subset of MSD instances – 15th June 2017 (Resolved)

Yesterday at 15:40 our monitoring services identified an issue with degraded performance on 1 of our Hypervisors hosting a small subset of MSD instances.

During the first 20 minutes yesterday we quickly identified the root cause and began notification of effected customers, whilst in parallel working to resolve the incident.

We had fully recovered at 16:30 and all instances were back to normal.

We have since moved everything off the offending hypervisor as a precaution so that we can now look at ensuring the effected hypervisors stability before its used again.

The root cause of the problem was a disk failure in the raid array and even though this should not have been an issue (we loose a disk in 1 of our many hypervisors on average once every 2 months) it should have used the hot standby and carried on (Infact we lost a disk in this very hyperrvisor at the start of this week, and customers would have been unaware of any issue).

Unfortunately the new disk seems to have had manufacturing defects which meant that it became unusable pretty quickly and because it hadnt finished its rebuild (from the previous disk failure) the hot standbys were not available.

Going forward we have implemented a number of checks to ensure that a disk is error free and ran in prior to being added to the array.

Nov 10 2016

Issue with connecting to MSD and Hornbill via Virgin Media ISP

Some customers may be experiencing issues this morning. This is due to Virgin Media issues in the south of England (Our data centres use both BT and Virgin so are still accessible, however reverse connections and access to instances may be effected if Virgin is your ISP).

Unfortunately this is somewhat outside of our control, however we may still be able to assist by changing your Authentication type if you can access your instance, so please contact cloudsupport@hornbill.com if you are effected.

Update : Virgin Have confirmed that this is an issue at Bromley UK and network engineers have been dispatched with replacement cards to rectify issue. Next update from Virgin is expected at 11AM.

Update : Virgin have started to redirect routes around the effected switch and customers should now be able to access. If you are still having issues and are a virgin customer we recommend you contact virgin support.

Oct 31 2016

Dirty COW

Here at Hornbill we take security very seriously, so when we received notification of CVE-2016-5195 which has been dubbed Dirty COW (Change on Write), we investigated our systems in order to identify if any of our servers were affected by this bug. The bug allows an attacker to potentially escalate a local account via gaining write access to read-only memory to that of root. This issue has existed in linux for nearly 10 years but has only recently become easier to exploit.

Inorder to exploit the bug and have an effect an attacker would need to be able to execute their own code on our servers. Whilst we fully believe this to be impossible (Our servers are never exposed to the outside world except for HTTPS\FTPS and then all the locations that are exposed have strict control including no code execution) we have taken action to patch all our linux servers and this process was completed earlier this morning.

For further information concerning Dirty Cow please see – https://dirtycow.ninja/

Regards,

Hornbill Cloud Support

Jul 20 2016

BT Issues effecting some customers

Some customers may be experiencing issues this morning. This is due to BT issues in the south of England (Our data centres use both BT and Virgin so are still accessible, however reverse connections and access to instances may be effected if BT is your ISP).

Unfortunately this is somewhat outside of our control, however we may still be able to assist by changing your Authentication type, so please contact cloudsupport@hornbill.com if you are effected.

BBC News report on the Issue – http://tinyurl.com/z7v24to

Mar 27 2015

Telephone Issues – 27th March 2015

We are currently experiencing issues with calls inbound to our offices due to an ISP issue. This is not causing any service issue and our emergency line is still available. If you require any assistance please email cloudsupport@hornbill.com rather than phone and we will contact you.

Aug 21 2014

Slow Performance – All UK Instances 21/08/2014 @ 09:10

Due to an unexpected failure of a fibre link to our data centre that hosts a number of UK instances, we are currently experiencing slower than expected performance. The root cause has now been identified as an issue with 1 of the fibre links to our data centre being down, our data centre networks team are currently reviewing traffic flows over the remaining links and migrating traffic to the most appropriate paths to maintain network stability, during this time you may experience some latency whist we move traffic flows around the network.

Update 10:00
Analysis has confirmed 1 effected core fibre is causing significant issues and all traffic is being rerouted to avoid the offending area. At this time no ETA for resolve and we are reviewing our emergency plan (Decision will be made in next hour as to whether we fall over to DR site).

Update 11:00
The issue is still being resolved (Ping times are slowly being reduced as the offending connection is circumvented).
We have reviewed our emergency plan and at this time believe that the inconvenience of moving instances to our secondary data centre would be worse than the slow performance (Would require at least 1 hour of Down time, possible changes to customers firewalls to allow connection to the new hosts and until the DNS propagated analysts\customers would need to use IP rather than Names).

We will continue to monitor this and update you on an hourly basis until resolved.

Update 12:00
The root cause has now been confirmed as a cable cut during road works on the A272 and engineers have been dispatched to resolve the issue. All traffic is therefore being redirected via secondary cables, which is causing load issues (This is also under investigation as secondary cables should be capable of providing full service).

We still believe that falling over to our secondary data centre is not required, however we will perform this on request (please contact Cloud Support Team to request migration).

We will continue to monitor this and update you on an hourly basis until resolved.

Update 13:00
Traffic is still being redirected, however over the last hour Ping times have dropped significantly (and therefore performance increased). We expect this to continue over the next hour and depending on route taken customers may already see the issue to be resolved.

We will continue to monitor this and update you on an hourly basis until resolved.

Update 14:00
Traffic for the most part is now redirected with Ping times < 10ms once again. We therefore believe that continued mitigation of the issue by routing will be enough to ensure that we do not need to switch to our secondary data centre.

Engineers have arrived on site at the cut fibre and are working to full restore communications.

We will continue to monitor this and update you on an hourly basis until resolved.

Update 15:00
All monitoring now shows that traffic\performance has been restored to pre-issue levels resolved by rerouting traffic. These changes will remain in place until the fibre is repaired. We do not expect any further issues with performance, however we will continue to monitor closely.

We will provide a full report to all customers within 24 hours.

Final Update 07:00 22/08
Our data centre team inform us that the effected cable was repaired (2KM of replacement fibre added) at 05:45 this morning and all traffic is following the original routes.

Jun 27 2014

UK Myservicedesk.com partial outage (27 June 2014, 10:15am)

by admin in Blog

This morning at 10:15am our UK data-centre came under attack from a sophisticated DDOS attack. Our Myservicedesk.com were not the target of these attacks but were on the same network as the target. This meant that traffic to our servers was being intermittently blocked which would have affected some of our users.

Our cloud support team reacted immediately, and the following is a timeline of the events as they unfolded:

10:15 – Our monitoring of servers indicated Slow Ping (average 120ms rather than 20ms) and packet loss (1 in 5)
10:17 – We contacted our data centre to report issue
10:19 – Communication sent to all effected customers
10:20 – We confirmed that a DDOS attack was under-way
10:22 – Our data centre started NULL routing the offending traffic
10:25 – Issue started to clear.
10:31 – Attack fully mitigated and all services returned to normal.

Now the issue has been resolved we will be working with our data-centre to ensure that steps are taken to avoid this in future (Including requesting that the targeted server be removed from the same network segment that hosts your instance).

Jun 23 2014

UK Myservicedesk.com partial outage (23 June 2014, 17:15pm)

by admin in Blog

At 17:15 we started experiencing intermittent network connectivity inside our cloud infrastructure that would have caused service disruption for a small number of our Myservicedesk.com customers. We began investigating the issue immediately, and full service was restored by 18:05.

Our initial diagnosis is that a network error occurred within our data centre providers network, and this disrupted communication between two critical parts of our cloud infrastructure. We have asked them to carry out a full investigation, and will pass these details on to customers as soon as they become available to us.

As always we aim to provide as little downtime as possible, and take outages like this very seriously. We will use any information gathered to further improve our cloud infrastructure and further improve it’s resilience.

Apr 10 2014

OpenSSL – Heartbleed Notice

Here at Hornbill we take security very seriously, so when we received notification of CVE-2014-0160 which has been dubbed heartbleed. we investigated our systems in order to identify if any of our servers were affected by this bug. The bug allows and attacker to read segments of memory allocated by OpenSSL making it possible to read the contents of private certificates we use to secure data in motion using SSL.

A small number of our front end web servers (which do not themselves hold any client data) were using the specific version of OpenSSL that is affected and therefore were vulnerable to this attack. We have already patched and tested to confirm that they were no longer vulnerable using http://filippo.io/Heartbleed/

We have reviewed our own logs and can see no evidence that this vulnerability has been exploited on our systems and the changes we have made to patch this will not affect our customers in anyway.  As it turned out, most of our production servers where not vulnerable as they are running stable versions of OpenSSL that are not impacted by this bug which was introduced in OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable, you can read all about the problem and get more details here: http://heartbleed.com/

Regards,

Hornbill Cloud Support

Accreditations

IS588876

Uptime SLA

99.5%

Uptime Target

99.95%

Platform Uptime Last Month
Platform Uptime This Month