Contractor is doing some electrical work - they did promise there would be NO power outage, and hence didn't even inform us a specific time a power outage could happen.
As we can see, they failed on this.
We will be on site to check out everything is in working order - a subsection of servers did not boot automatically.
We run a completely lights out facility so there isn't normally a person on standby and as of yet we are unsure what has happened. We are hoping it is just a blown fuse due to sudden spike on load, tho our servers are setup for staggered boot.
UPDATE 1
Couple switches had their automatic fuses blown - easy fix.
Several servers apparently simply crashed.
The reason for outage was that they had to move the whole building to a generator, they had matches the phases so that they could feed from the generator & mains simultaneously, but apparently forgot to increase the RPM of the generator upfront and the peak was simply too large for the generator to ramp up fast enough.
Move back to main power went smoothly and without an issue.
We are now checking which servers are down and manually rebooting and checking them out.
UPDATE 2
Everything checks out and all nodes should be up & running normally.
Lørdag, November 28, 2015
