We are checking that all systems are operational right now.
UPDATE: Most things started automatically, some backend stuff needed to be checked. Things are updating right now, but by the looks of it almost everything recovered automatically as expected.
UPDATE 2: Few nodes need manual attention one by one, we are going through these right now.
UPDATE 3: Last 5 nodes are being investigated why they did not reboot normally. Everything else is online. Only 1 drive failure and that was on a backend system so far. Everything else worked fine, mostly of peculiar firmware issues which required hard cycling tho.
UPDATE 4 - CONCLUSION:
There was a major electric grid outage in Helsinki today -- the whole grid went down. Consequently, datacenters, servers etc. here and there went down as well.
This caused outages for our Helsinki DC as well today. Majority of services autorecovered as expected. Few did not.
Hardware failures were zero, that one backend drive was already failed for awhile.
This was more arduous recovery than usual however, typically it has been less work. Even some brand new servers have this nasty firmware bug they don't fully recover from power outage, or voltage fluctuations.
Another segment was software issues, one VM host server had misconfigured arrays (and never had prior downtime, config error during initial setup, finished by 2nd error: never testing recovery), and some manual filesystem checks had to be done.
There might be some of our dedicated server customers who might still need help, but monitoring of those are beyond our purview and we have no means to determine until a ticket is opened. Many dedicated server customers block ICMP ECHO needlessly meaning we cannot monitor them at all. Please open a ticket if yours is still down.
We will be shortly responding to tickets still pending, now that the actual outages has been resolved.
Sorry for the inconvenience.
As a reminder, there is also scheduled power & electrical maintenance this september: https://pulsedmedia.com/clients/announcements.php?id=599
Best Regards,
Pulsed Media Staff
Thursday, August 3, 2023
