There was an electrical issue at the building electrical cabinet. Took a little bit of time for electrical contractor to arrive during Holiday sunday-monday night.
At roughly 01:45AM local time one of the phases went down on one of the electrical feeds we have. This caused a cascade on cooling side since 3-phase chillers need all 3-phases.
The issue was quickly isolated and found by roughly 02:30AM and building staff on alert was notified. Sometime around 05:10AM servers started to come back online.
Root cause has been isolated and steps are being taken to ensure this does not happen again.
The issue has now been fixed, and almost all hardware is up and running.
There are a couple of servers still which needs more attention but might take an day or two extra, as well staff is exhausted after getting an middle of night alert.
The servers known with remaining issues are: sentnel, walton, greenwood, dumper
Monday, June 27, 2022
