Helsinki DC has an outgoing outage, and we are investigating the situation right now.
We will update this announcement as soon as we know more.
**UPDATE 1** Transformer maintenance on the premises. Premises failed to inform us about this despite this was known from last fall. Staff on site, we are working to see if we can restore power to a number of servers. Outage from premises manager was told to be maximum 10 hours. After power is restored it will take a moment to get everything back online again and for the UPS batteries to recharge.
**UPDATE 2 Saturday 00:50** Premises manager failed to notify us of this. The transformer maintenance began at 23:00 Friday Night, and is supposed to last maximum of 10 hours. Since they did not notify us we could not arrange a generator on site either, and being weekend night it's quite hard to get a generator on site either, it should've been arranged no later than 18:00 Friday. They dropped the ball completely on this, other datacenters on the same premises were notified but not us. These premises contain a lot of datacenters of many sizes and from companies of many nationalities, to best of our knowledge we are the only ones who have not been informed about this. Rest assured, some very serious chats with the premises manager will ensue from this.
No battery backup (currently) in the world can last for 10 hours for a datacenter. So once our batteries ran out, so did the servers.
We have staff on site, and getting well caffeinated for checking that everything gets back online normally. Once we either get power from someone else's generator or our own generator somehow miraculously on the site we will start restoring services as quickly as possible. We estimate that most servers will come back online without a hitch, but there will likely be a few last services lingering until saturday night for full restoration.
**UPDATE 3 Saturday 01:11** We got the word that power might be restored as quickly as just 1-1½ hours.
**UPDATE 4 Saturday 01:59** Partial power restored. Routing restored
**UPDATE 5 Saturday 02:18** All power restored, we are now checking all servers startup as expected. Battery banks are recharging.
**UPDATE 6 Saturday 02:47** Almost all servers back online. A few remain to get back online, less than 50 now.
**UPDATE 7 Saturday 04:36** Every single server should be up and running. Please open a ticket if you are still encountering issues.
Friday, March 19, 2021
