Storage Performance Degradation

Resolved

We are continuing to improve performance of the storage infrastructure, though we are happy to report that all services are now able to operate normally and no sites should be experiencing any issues at this time.

If you do face any ongoing issue, please do get in touch as it is likely to be unrelated to the events from last night.

Once again, our sincere apologies for the inconvenience that you have faced.

Monitoring

Our engineering team has now largely resolved the performance degradation within our storage cluster, and all sites/services should now be operating at near normal performance levels or all clients. We sincerely apologise for any inconvenience these performance issues have caused.

Our team continues to work on the platform to complete repairs, and we anticipate these will complete over the coming few hours. We will of course continue to monitor services closely in the meantime.

Monitoring

Just to advise that further improvements have been made as we work to restore regular service to our storage platform. At this time all shared/reseller servers should be experiencing close to regular performance, with all services operational.

Our engineers are still working on this, with further improvements and a final resolution anticipated over the coming few hours.

Monitoring

Service is largely restored on the vast majority of servers and most websites are now responding, though we’re still operating at reduced I/O capacity. We are continuing to monitor and make improvements where possible.

Assessed

We are experiencing severe service issues on a number of servers, which is caused by a degradation of performance on the block storage. In order to prevent a larger issue, our engineers have been forced to limit the speed at which block storage operates.

Though this has successfully mitigated a wider problem, for servers which are used to operating at much higher speeds, they are experiencing extremely high loads due to insufficient I/O performance. Although services are running, many websites, notably dynamic websites, are not functional. Static websites or websites served by cache are typically responding.

We are continuing to work on returning service to normal as quickly as possible. Our apologies for the inconvenience this evening.

5 Affected Services: