Today's downtime has been solved
Today's attack was an rather old method: Mail bomb.
Curious part is that how it passed google filters, as we use for support emails and such google for proper spam filtration. Not only did it pass Google's filters, search could find the threads but in-built custom filters didn't? Thousands upon thousands upon thousands of e-mails in very short period and neither Google nor Yandex had any filtration for such an occurence.
Further it stuns me how WHMCS lacks any protection what-so-ever too, WHMCS not only checks on these bombed mails, then sends a reply, but POP processes keeps on piling up: There is no locks, there is no "alive"/"already running" methods, nothing to check if the e-mail import job is already running nor even rudimentary filtration, ie. threshold of e-mails from single address/domain/ip.
We'll need to code our own layer on top of WHMCS e-mail import to add these checks to avoid those processes from piling up.
The end result of this mail bombing was that the system was overwhelmed with the quantity of processes, which in their turn overwhelmed mysql. All the errors and e-mails then resulted in system partition filling up and further stopping from anything working.
What stuns us most is that this 90s style attack can still work today, with all the work against spam, with all the tools available to us, it still gets through and is viable attack vector. To stop things like this we specifically chose Google as the e-mail host for pulsedmedia.com domain accounts, thinking their filtration system is so excellent and they have a lot of hardware online to stop even the largest attacks of this kind. This attack was rigged for exponential growth by solely utilizing yandex, google and our resources.
The situation is now alleviated and some failsafes has ALREADY been implemented, but more will be implemented in near future.
We are very sorry for the inconvenience this may have caused to you.
Monday, July 30, 2012
