One top of the rack switch has failed causing several dozen servers to be down at the moment.Staff is en route to datacenter. ETA for full resolution 3hours.UPDATE 23:02 - Resolved, all servers back online. A Rittal PDU had failed. This is a known issue with this model of PDU, their fusing over time simply fails and cannot be reset anymore. ... Read More »
There's been a big power outage in the whole Helsinki area energy grid.This has caused several backend infrastructure servers and production servers to go down and into "halted" state where not even remote management responds.Staff en route to datacenter.Update 10:38 - Some of the infrastructure servers brought back online -- getting servers hard ... Read More »
As part of the removing the faulty aggregation switch many top of the rack switches got also doubled bandwidth as part of the replacement, and were already planned.Multiple racks got uplinks doubled today.We do not expect to see a dramatic difference in total bandwidth utilization, but on some rare cases this might stabilize throughput speeds for ... Read More »
Mysteriously link aggregation from that old aggregation switch stopped working, despite media functional.This aggregation switch was already planned to be removed regardless, and only a few next tier switches were still behind this aggregation layer.Intervention is scheduled for tomorrow, and expected to be fixed within 24hours.In the meantime, ... Read More »
