A significant number of servers are down due to a unknown switch issue. We are working on it right now.
Sorry for something like this to happen at such an unfortunate time; Middle of new years celebrations!
Update 1 (00:35): All switches responsive and rebooted, issue is not with swtiches. 68 nodes are afflicted. On site intervention scheduled, current estimated full resolution 1hr 45minutes.
Update 2 (01:11): Estimated recovery time 1hr 15minutes. Suspected Denial of service attack exploiting hardware weakness, relating to august issues.
Update 3 (03:51): 12 nodes still down, working on them.
Update 4 (04:29): Resolved. All nodes up and running. relates to August issues. Diagnosis still undergoing.
Sorry for the downtime! All servers are currently response and signals good status without issues.
Diagnosis will continue for quite a while after the holidays. It would be highly unlikely for this to be co-incidence in regards of near perfect timing to maximize downtime; this number of servers simply do not crash without a reason all of sudden.
Tuesday, January 1, 2019