There's been a big power outage in the whole Helsinki area energy grid.This has caused several backend infrastructure servers and production servers to go down and into "halted" state where not even remote management responds.Staff en route to datacenter.Update 10:38 - Some of the infrastructure servers brought back online -- getting servers hard ... Read More »
As part of the removing the faulty aggregation switch many top of the rack switches got also doubled bandwidth as part of the replacement, and were already planned.Multiple racks got uplinks doubled today.We do not expect to see a dramatic difference in total bandwidth utilization, but on some rare cases this might stabilize throughput speeds for ... Read More »
Mysteriously link aggregation from that old aggregation switch stopped working, despite media functional.This aggregation switch was already planned to be removed regardless, and only a few next tier switches were still behind this aggregation layer.Intervention is scheduled for tomorrow, and expected to be fixed within 24hours.In the meantime, ... Read More »
One of the switches for 10G servers is down, intervention is being scheduled to get it back to working order.This is a new type of crash, it still responds just enough not to trigger typical test alerts, monitoring still shows it as being online. But alas it does not respond to management and servers under it does not respond. Well the servers ... Read More »
