This morning at 10:15am our UK data-centre came under attack from a sophisticated DDOS attack. Our Myservicedesk.com were not the target of these attacks but were on the same network as the target. This meant that traffic to our servers was being intermittently blocked which would have affected some of our users.
Our cloud support team reacted immediately, and the following is a timeline of the events as they unfolded:
10:15 – Our monitoring of servers indicated Slow Ping (average 120ms rather than 20ms) and packet loss (1 in 5)
10:17 – We contacted our data centre to report issue
10:19 – Communication sent to all effected customers
10:20 – We confirmed that a DDOS attack was under-way
10:22 – Our data centre started NULL routing the offending traffic
10:25 – Issue started to clear.
10:31 – Attack fully mitigated and all services returned to normal.
Now the issue has been resolved we will be working with our data-centre to ensure that steps are taken to avoid this in future (Including requesting that the targeted server be removed from the same network segment that hosts your instance).
At 17:15 we started experiencing intermittent network connectivity inside our cloud infrastructure that would have caused service disruption for a small number of our Myservicedesk.com customers. We began investigating the issue immediately, and full service was restored by 18:05.
Our initial diagnosis is that a network error occurred within our data centre providers network, and this disrupted communication between two critical parts of our cloud infrastructure. We have asked them to carry out a full investigation, and will pass these details on to customers as soon as they become available to us.
As always we aim to provide as little downtime as possible, and take outages like this very seriously. We will use any information gathered to further improve our cloud infrastructure and further improve it’s resilience.