Hornbill Trust
Call Us: +44(0)208 582 8282

Social: Facebook, Google+

Nov 10 2016

Issue with connecting to MSD and Hornbill via Virgin Media ISP

Some customers may be experiencing issues this morning. This is due to Virgin Media issues in the south of England (Our data centres use both BT and Virgin so are still accessible, however reverse connections and access to instances may be effected if Virgin is your ISP).

Unfortunately this is somewhat outside of our control, however we may still be able to assist by changing your Authentication type if you can access your instance, so please contact cloudsupport@hornbill.com if you are effected.

Update : Virgin Have confirmed that this is an issue at Bromley UK and network engineers have been dispatched with replacement cards to rectify issue. Next update from Virgin is expected at 11AM.

Update : Virgin have started to redirect routes around the effected switch and customers should now be able to access. If you are still having issues and are a virgin customer we recommend you contact virgin support.

Oct 31 2016

Dirty COW

Here at Hornbill we take security very seriously, so when we received notification of CVE-2016-5195 which has been dubbed Dirty COW (Change on Write), we investigated our systems in order to identify if any of our servers were affected by this bug. The bug allows an attacker to potentially escalate a local account via gaining write access to read-only memory to that of root. This issue has existed in linux for nearly 10 years but has only recently become easier to exploit.

Inorder to exploit the bug and have an effect an attacker would need to be able to execute their own code on our servers. Whilst we fully believe this to be impossible (Our servers are never exposed to the outside world except for HTTPS\FTPS and then all the locations that are exposed have strict control including no code execution) we have taken action to patch all our linux servers and this process was completed earlier this morning.

For further information concerning Dirty Cow please see – https://dirtycow.ninja/

Regards,

Hornbill Cloud Support

Jul 20 2016

BT Issues effecting some customers

Some customers may be experiencing issues this morning. This is due to BT issues in the south of England (Our data centres use both BT and Virgin so are still accessible, however reverse connections and access to instances may be effected if BT is your ISP).

Unfortunately this is somewhat outside of our control, however we may still be able to assist by changing your Authentication type, so please contact cloudsupport@hornbill.com if you are effected.

BBC News report on the Issue – http://tinyurl.com/z7v24to

Mar 27 2015

Telephone Issues – 27th March 2015

We are currently experiencing issues with calls inbound to our offices due to an ISP issue. This is not causing any service issue and our emergency line is still available. If you require any assistance please email cloudsupport@hornbill.com rather than phone and we will contact you.

Aug 21 2014

Slow Performance – All UK Instances 21/08/2014 @ 09:10

Due to an unexpected failure of a fibre link to our data centre that hosts a number of UK instances, we are currently experiencing slower than expected performance. The root cause has now been identified as an issue with 1 of the fibre links to our data centre being down, our data centre networks team are currently reviewing traffic flows over the remaining links and migrating traffic to the most appropriate paths to maintain network stability, during this time you may experience some latency whist we move traffic flows around the network.

Update 10:00
Analysis has confirmed 1 effected core fibre is causing significant issues and all traffic is being rerouted to avoid the offending area. At this time no ETA for resolve and we are reviewing our emergency plan (Decision will be made in next hour as to whether we fall over to DR site).

Update 11:00
The issue is still being resolved (Ping times are slowly being reduced as the offending connection is circumvented).
We have reviewed our emergency plan and at this time believe that the inconvenience of moving instances to our secondary data centre would be worse than the slow performance (Would require at least 1 hour of Down time, possible changes to customers firewalls to allow connection to the new hosts and until the DNS propagated analysts\customers would need to use IP rather than Names).

We will continue to monitor this and update you on an hourly basis until resolved.

Update 12:00
The root cause has now been confirmed as a cable cut during road works on the A272 and engineers have been dispatched to resolve the issue. All traffic is therefore being redirected via secondary cables, which is causing load issues (This is also under investigation as secondary cables should be capable of providing full service).

We still believe that falling over to our secondary data centre is not required, however we will perform this on request (please contact Cloud Support Team to request migration).

We will continue to monitor this and update you on an hourly basis until resolved.

Update 13:00
Traffic is still being redirected, however over the last hour Ping times have dropped significantly (and therefore performance increased). We expect this to continue over the next hour and depending on route taken customers may already see the issue to be resolved.

We will continue to monitor this and update you on an hourly basis until resolved.

Update 14:00
Traffic for the most part is now redirected with Ping times < 10ms once again. We therefore believe that continued mitigation of the issue by routing will be enough to ensure that we do not need to switch to our secondary data centre.

Engineers have arrived on site at the cut fibre and are working to full restore communications.

We will continue to monitor this and update you on an hourly basis until resolved.

Update 15:00
All monitoring now shows that traffic\performance has been restored to pre-issue levels resolved by rerouting traffic. These changes will remain in place until the fibre is repaired. We do not expect any further issues with performance, however we will continue to monitor closely.

We will provide a full report to all customers within 24 hours.

Final Update 07:00 22/08
Our data centre team inform us that the effected cable was repaired (2KM of replacement fibre added) at 05:45 this morning and all traffic is following the original routes.

Jun 27 2014

UK Myservicedesk.com partial outage (27 June 2014, 10:15am)

by admin in Blog

This morning at 10:15am our UK data-centre came under attack from a sophisticated DDOS attack. Our Myservicedesk.com were not the target of these attacks but were on the same network as the target. This meant that traffic to our servers was being intermittently blocked which would have affected some of our users.

Our cloud support team reacted immediately, and the following is a timeline of the events as they unfolded:

10:15 – Our monitoring of servers indicated Slow Ping (average 120ms rather than 20ms) and packet loss (1 in 5)
10:17 – We contacted our data centre to report issue
10:19 – Communication sent to all effected customers
10:20 – We confirmed that a DDOS attack was under-way
10:22 – Our data centre started NULL routing the offending traffic
10:25 – Issue started to clear.
10:31 – Attack fully mitigated and all services returned to normal.

Now the issue has been resolved we will be working with our data-centre to ensure that steps are taken to avoid this in future (Including requesting that the targeted server be removed from the same network segment that hosts your instance).

Jun 23 2014

UK Myservicedesk.com partial outage (23 June 2014, 17:15pm)

by admin in Blog

At 17:15 we started experiencing intermittent network connectivity inside our cloud infrastructure that would have caused service disruption for a small number of our Myservicedesk.com customers. We began investigating the issue immediately, and full service was restored by 18:05.

Our initial diagnosis is that a network error occurred within our data centre providers network, and this disrupted communication between two critical parts of our cloud infrastructure. We have asked them to carry out a full investigation, and will pass these details on to customers as soon as they become available to us.

As always we aim to provide as little downtime as possible, and take outages like this very seriously. We will use any information gathered to further improve our cloud infrastructure and further improve it’s resilience.

Apr 10 2014

OpenSSL – Heartbleed Notice

Here at Hornbill we take security very seriously, so when we received notification of CVE-2014-0160 which has been dubbed heartbleed. we investigated our systems in order to identify if any of our servers were affected by this bug. The bug allows and attacker to read segments of memory allocated by OpenSSL making it possible to read the contents of private certificates we use to secure data in motion using SSL.

A small number of our front end web servers (which do not themselves hold any client data) were using the specific version of OpenSSL that is affected and therefore were vulnerable to this attack. We have already patched and tested to confirm that they were no longer vulnerable using http://filippo.io/Heartbleed/

We have reviewed our own logs and can see no evidence that this vulnerability has been exploited on our systems and the changes we have made to patch this will not affect our customers in anyway.  As it turned out, most of our production servers where not vulnerable as they are running stable versions of OpenSSL that are not impacted by this bug which was introduced in OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable, you can read all about the problem and get more details here: http://heartbleed.com/

Regards,

Hornbill Cloud Support

Jan 24 2014

UK Myservicedesk.com outage (24 January 2014, 12 55pm)

We are experiencing some network issues at our UK data center in Portsmouth. The Cloud Team are investigating the issue and we will post an update here as soon as possible.

update: 13:53 – Some of the UK Myservicedesk.com are starting to come back on line. The Cloud Team are working on restoring the remaining systems and we will provide an update here as soon as possible.

update 14:29 – Almost all of the UK Myservicedesk.com instances are now back online. The Cloud Team are performing checks to ensure that there are no database corruptions. We will provide another report as soon as
possible.

update 14:36 – all affected Myservicedesk.com instances were back online at 14:35. Customers have been called
and a summary of the outage will be sent to all primary contacts on 27th January.

Infrastructure Team

 

 

 

Dec 17 2013

Outage at UK datacenter (17 December 2013, 17:09)

At approximately 17:09, we experienced an outage at our primary datacenter which has affected some Myservicedesk.com instacnes in the UK. We are working on resolving the problem and will post an update here shortly. If you have any questions, please call the Cloud Support number on +44 203 286 0582

Update: 17 December ,19:01: Affected services recovered around 19:01. We will send a report on the outage
to primary contacts tomorrow morning.

Systems will be monitored as always by our 24×7 On Call team. If you have any questions or concerns, please
email cloudsupport@hornbill.com

Thank you and apologies for any inconvenience caused

Infrastructure Team

 

Accreditations

IS588876

Uptime SLA

99.5%

Uptime Target

99.95%

Platform Uptime Last Month
Platform Uptime This Month