istgt had crashed and forced a reboot (only way to fix istgt connection failures!), which resulted in some ZFS regressions.
Letting the storage to resilver for a while before onlining everything again.
Work has begun to bring solo1 online.
UPDATE
Jabba2 array1 decided to resync as well, ETA for completion at current speed is about 5 hours.
However, we will just increase min sync speed and online things far before that.
UPDATE2
Solo1 is building it's array, ETA ~7hrs
Jabba2: We are using Qlogic/Dell NetXen card for it's 10Gb, and a driver issue makes it unavailable every now and then on reboot, took a while to recognize that it's yet again failing to load and that's making us take longer than expected in bringing things back online.
Jabba1: Solo1 will be imported to Jabba1 for migrations, we are unsure will we update the OS images for new IP or will we keep it like it is, going through Jabba1 for a while. Probably the latter for first couple of weeks.
UPDATE 3
Some nodes are up, for some reason nodes from Jabba1 ZFS pool are not getting online, and we are seeing some worrying things from that pool (3 disks resilvering while parity 2??). We'll know more shortly, might be just so badly overloaded due to the resilvering process.
Solo1 Array1: very huge variance in disk performance, swapping 3 of the 5 disks in an attempt to get more similar performance drives in the array.
Solo2: Being installed.
Jabba2 Array1: Still resyncing (resilvering), with limited speed, all nodes utilizing that are up & running tho.
Jabba2 Array2: Is doing initial sync (5x2Tb).
Recap: Storage entering online is two 5x3Tb arrays and one 5x2Tb array.
Solo3 might get built today as well (5x3Tb array)
UPDATE 4
We are manually running fsck on those machines which did not boot automatically - some of them have some corruption but so far every single one has booted up fine after doing manual fsck.
We are soon fully back in production.
Migrations will begin by tomorrow on the newly built arrays.
Wednesday, August 7, 2013
