One top of the rack switch has failed causing several dozen servers to be down at the moment.
Staff is en route to datacenter.
ETA for full resolution 3hours.
UPDATE 23:02 - Resolved, all servers back online. A Rittal PDU had failed. This is a known issue with this model of PDU, their fusing over time simply fails and cannot be reset anymore. Failure rate is low enough however not to outright replace all units in already in use.
Sunday, August 22, 2021
