A lot of servers has already been restored.
All dedicated should be up and running. All VPS if not already running should be running within next hour. No danger ever occured for any dedicated servers. VPS nodes are safe as well.
Most seedbox servers are quickly restored, but there are some which will require a lot more work and data loss is likely. For those nodes we expect turnaround to be a few days, and all users will be fully compensated as per our SLA policy at http://wiki.pulsedmedia.com/index.php/Pulsed_Media_Seedbox_SLA_Policy but in this case we will give +2 days extra automatically, as it's clerical error for not being paranoid enough about ex-employees.
Please note that this happened because of a gruntled ex-employee. All his access was revoked late last week, but he apparently had a private copy of a server password database for bulk of Seedbox servers. His online access had already been revoked to this said database.
As far as we can tell by logs and crawling through data on the servers only action taken is to change password on bulk of servers, and a few has been noted that "rm -rf /" was executed. This ex-employee has been noted to be bragging about this action in several forums.
Lesson learned, in future no employee shall be granted full server access unless a special case warrants it, and only in a way access can be revoked quickly. Personally, i implicitly trusted this person and have known him for roughly 11years so i was fully expecting that professionally we can part our ways gracefully.
Dedicated server passwords are a separate database, so he did not have access to them. Same goes for VPS nodes.
If your dedicated server is down for some reason, please do not hesitate to open a ticket. All VPS instances should be up & running within the hour if not operational already. Dedicated & VPS were brought to rescue mode only because of a too greedy filtering on automated rescue mode change.
Several electrical and hard drive failures were detected on this, and DC is already working on them, if not fixed already.
Sorry for all the inconvenience caused,
Aleksi
Pulsed Media
Monday, November 21, 2011
