A series of successive lightning strikes in Belgium last Thursday managed to knock some cloud storage systems offline briefly, causing errors for some users of Google’s cloud infrastructure services. The affected disks were part of Google Computer Engine (GCE), a utility that lets people run virtual computers in the cloud on Google’s servers.
Advertisement
In its incident report, Google says that less than 0.000001% of its Western Europe permanent disk space was lost, but because the data was replicated across multiple servers, customers did not permanently lose data. There was no effect on Standard Persistent Disks created after 09:19.
“Google Compute Engine instances and Persistent Disks within a zone exist in a single Google data center and are therefore unavoidably vulnerable to data center-scale disasters”, the company said. “We have conducted a thorough analysis of the issue, in which we identified several contributory factors across the full range of our hardware and software technology stack, and we are working to improve these to maximise the reliability of GCE’s [Google Compute Engine] whole storage layer”. According to the search giant, research is already underway to improve cache data retention and response procedures for system engineers.
The statement read: “Google takes availability very seriously, and the durability of storage is our highest priority”. But some hardware which is more susceptible to power failure suffered irreversible data loss.
Google said it was “wholly responsible” for the outage, and it urged affected customers to consider duplicating and storing their data through other services that it provides. And when lightning hits a building multiple times, it doesn’t need to hit in exactly the same spot each time to cause extra damage. Gale explained that power or phone cables located at a distance from a building can still cause disruptions if struck by lightning.
Advertisement
Google continues to state that this has led to “permanent data loss” on some drives holding the most recently changed files. “If you get four large strikes it wouldn’t surprise me that it has affected the facility”. Most of the storage systems were connected to battery backups and survived the power outage that followed the lightning strikes.
