Final stretch is being approached. Currently mainly waiting for providers to see their support options to do in batches easier.
Only remaining nodes are those where data loss occured. Most of those cases customer data is completely intact and it's only OS files which are missing.
On other nodes we are waiting for IPMI fixes and such.
If we have any luck this should be over by tonight, leaving maximum downtime to 36hrs to those on the "worst case scenario" servers. but do not count on that, because there is still a lot of work to do and a lot of things to be found out and coordinating with multiple different DCs doesn't make things simpler.
Of course, a lot of double checking will still remain, as we don't need to just double check, but triple check that servers are in good production capacity after restoration.
Really sorry about this whole ordeal. Be assured that there will be in future great emphasis not to give this much control again to any single employee.
Tuesday, November 22, 2011
