UPDATE: This does not concern dedicated servers, only shared seedbox servers. All dedicated servers are untouched, some only briefly put into rescue mode by overzealous filtering on automation.
For some servers "rm -rf /" was executed, fortunately most of the time this didn't have time to complete even part way, so restoration without data loss is possible, if not almost trivial in some cases.
Further, we have just outsourced some of that work to a reputable 3rd party company to speedup the process. They are right now looking into how they can do direct in-place ext4 undelete in the rescue mode environment we have. Let's hope they manage to do that.
Last option is to "rsync" new operating system files in place, and then manually verify working condition.
Current estimate is that only in couple servers actual customer data loss occured, from which some were actual hardware hdd failures brought into light on the reboot. We are hoping even on those couple servers we can restore all data.
However, on these servers the restoration WILL take quite a bit of time as it's quite an intensive process, and thus are on lower priority than those we simply have to verify they are in perfect working condition.
The estimation for complete restoration for all nodes, for all types of seedbox services, regardless what provider is being used (some are harder to work with than others) remains at 3 days.
In a case where you have to wait more than 24hrs for restoration 3 days extra service time will be given, if it takes full 3 days you will be given 5 days extra service time. This is an extra 2 days on top of our normal SLA readable at: http://wiki.pulsedmedia.com/index.php/Pulsed_Media_Seedbox_SLA_Policy
On top of that we are now considering enacting full weekly backups on even the cheapest of SB plans on our under development storage cluster if we can make that financially feasible.
Monday, November 21, 2011
