This has caused several backend infrastructure servers and production servers to go down and into "halted" state where not even remote management responds.
Staff en route to datacenter.
Update 10:38 - Some of the infrastructure servers brought back online -- getting servers hard rebooted.
Update 10:59 - Certain types of servers essentially all had crashed into a state requiring physical power cycling. Waiting for servers to reboot, complete their filesystem checks etc. before we start checking servers one by one.
Update 11:20 - Almost all production servers back online, only a few remains.
Update 12:48 - Resolved. All production servers we can monitor are online. If your service is still down contact support. On dedis make sure that your server responds to ICMP Echo
Saturday, August 21, 2021
