Full recovery has been achieved in practice now. Only 2 servers remain to be restored, and both are more or less half way there. These both servers were problem cases.
Helpdesk backlog has been mostly dealt with as well. The past week resulted in more tickets & replies than a average month does. That was A LOT of tickets to handle. Despite that, average response times for the month remain on average level, however down from the exemplary response times prior to this week which were on record level. We will add several new metrics to measure from support which are shown to be vital from the recent events, such as highest, and lowest 5% times median. Only several tickets remain still to be solved from the current backlog.
Slight backlog on orders exists, but that too is generally less than 48hrs.
So i am very happy to say that by all practical means recovery is complete in big picture and we start getting back to regular schedule.
There will be slightly more things to handle for the next 2 weeks as well, as we go through all nodes again to verify them, and certainly there will be some issues from this massive operation on the nodes to be resolved. So support and provisioning schedules are a little bit prolonged, but not badly.
Our internal security practices will become even more paranoid, from an already strict, slightly paranoid level. That means a lot of development, but it's better to be safe than sorry, and experience shows that even people you implicitly trust cannot be trusted at any level.
Despite all of this, i'm hoping we can still push for christmas specials and new services during december despite development has been delayed several on those.
As always, i'm personally available at IRC shall any questions arise aside regular support. So join us at #PulsedMedia on Freenode. Just remember IRC really isn't a realtime medium, but i'm checking in multiple times per day there, so feel free to hang around and chat with other users :)
Best Regards,
Aleksi
Pulsed Media
Monday, November 28, 2011
