A fire in one of the OVH datacenters has destroyed one datacenter and knocked two others offline. It took 100 firefighters and 43 fire trucks to fight the fire in the five-story building. Even though the fire department was quick to respond, and the fire was brought under control relatively quickly, the impact has been big.
In a press statement OVH promised “to communicate as transparently as possible on the progress of our analyses and the implementation of solutions”.
OVH is the largest hosting provider in Europe and the third largest in the world. The cloud computing company provides virtual private servers, dedicated servers, and other web services.
Customers are being advised by the company to enact their disaster recovery plans after the fire has rendered multiple data centres unserviceable, impacting websites around the world, and a number of organisations involved in cybersecurity.
One such company, Acceis, met the situation with an admirable sense of humour, while providing a dramatic view of the fire.
Data and servers in the cloud
Many organizations use some type of cloud services to keep their setup flexible. But the old saying about the cloud that “it’s your data on someone else’s computer” hits home when you suddenly loose a big chunk of your server capacity or your web services out of the blue.
It’s too late to think about a backup plan when you find yourself needing one. As a result of this incident some customers of OVH state their web services are inaccessible. Which usually means that their websites are inaccessible as well.
Sadly, for video game maker Rust, the incident has led to a total data loss, leaving no way for recovery (although the company seems to be restoring services fairly rapidly).
BleepingComputer provided this list of victims:
“The list of impacted clients includes provider of free chess server Lichess.org, videogame maker Rust, cryptocurrency exchange Deribit’s blog and docs sites, telecom company AFR-IX, encryption utility VeraCrypt, news outlet eeNews Europe, the art building complex Centre Pompidou, and many others.”
And since the data centre site is off limits for now, it will take a while before the offline centres can be restarted.
OVH’s Octave Klaba tweeted:
“We plan to restart SBG1+SBG4+the network by Monday March,15 and SBG3 by Friday March,19.”
Pros and cons of the cloud
The fire is a very dramatic reminder that the cloud has a down side. As with all technology, there are pros and cons to using it.
The great advantages of the cloud are that it makes worrying about hardware somebody else’s problem, its scalable and flexible—it can react quickly to changes in demand and you pay for what you use—and it’s accessible from anywhere.
But even in the cloud your data is always somewhere, and that somewhere still needs security (which may be different from what you’re used to), data protection, internet access, backups and disaster recovery.
As OVH put it:
Customers should immediately bring into effect their disaster recovery plans as OVH is working on restoring its services.
That raises the question of how many of its customers had such a disaster recovery plan. It’s too late for them if they didn’t, but if you weren’t affected by this fire, now is the perfect time to check that you have one!
Update March 12
OVH has hinted that the fire was likely caused by a faulty UPS power supply. OVH’s founder Octave Klaba says in a video that the fire department’s thermic camera images suggests that two UPS were the source of the fire at the data center.
An uninterruptible power supply or uninterruptible power source (UPS) is an electrical apparatus that provides emergency power when the input power source or main power fails. UPS overheating generally happens due to insufficient ventilation, or due to overload. In both cases there will need to be an investigation into the underlying cause, but at least they will know where to start. Which is poor consolation for the organizations crippled by the fire and the subsequent loss of servers and data.
