storage maintenance over

sorry took a little bit longer than expected, did some additional hardware level maintenance at the go.

Servers are booting back online, and we are now targeting that only downtimes for the next several weeks are data pool migrations.

7th Aug 2013
caching addition -> short downtime max 15mins

we are adding SSD cache to one of the pools, this will cause a maximum of 15mins downtime.

7th Aug 2013
back online

all nodes were put to boot back online a few moments ago and failed disks replaced.

FS is giving for couple of nodes error of data corruption, but most likely fsck will handle that without data loss.

7th Aug 2013
espoo servers down

the initial storage pool server suffered yet another failure today - a extremely bad batch of drives now reaching 50% failure rate within the month.

the first pool is right now resilvering but FS is warning that data corruption may have occured, it will be brought back online shortly along with the nodes.

 

7th Aug 2013