A significant number of servers are down due to a unknown switch issue. We are working on it right now.
Sorry for something like this to happen at such an unfortunate time; Middle of new years celebrations!
Update 1 (00:35): All switches responsive and rebooted, issue is not with swtiches. 68 nodes are afflicted. On site intervention scheduled, current estimated full resolution 1hr 45minutes.
Update 2 (01:11): Estimated recovery time 1hr 15minutes. Suspected Denial of service attack exploiting hardware weakness, relating to august issues.
Update 3 (03:51): 12 nodes still down, working on them.
Update 4 (04:29): Resolved. All nodes up and running. relates to August issues. Diagnosis still undergoing.
Update 5:
Sorry for the downtime! All servers are currently response and signals good status without issues.
Diagnosis will continue for quite a while after the holidays. It would be highly unlikely for this to be co-incidence in regards of near perfect timing to maximize downtime; this number of servers simply do not crash without a reason all of sudden.
Tuesday, January 1, 2019
