Got complaints from my customers that their websites are going up & down like a yoyo. Checked, and that’s true enough. Although at first it seems okay, but when you hit reload, you got the error message.
Thank God for squid, so even when the server is having problems, it’s not instantly obvious to the visitors. Only to the admins. (hint: admin pages are usually set with no-cache header)
The error messages are related to mysql. So I checked it, and indeed MySQL was overloaded to the max. mysqladmin -h localhost -u root -p status showed that it’s handling crazy amount of queries per second.
I was a bit baffled. Normally, that will cause MySQL to fell straight away. But at that time, it’s “just” going up & down. So I looked for more clues using top.
I quickly noticed that there are huge numbers of smtpd processes. What’s going on ?
So I checked the mysql query log.
Turned out that the spammers are trying to brute force their way to my smtp server 🙁
They’re trying various combinations of username & password. They doesn’t seem to be anywhere successful, but they sure caused MySQL to act funny.
smtpd authenticate to a table in MySQL. The table is small, so it must be cached already by MySQL.
But even cached, when the requests are coming very rapidly, it’d still hurt.
Checked /etc/postfix/master.cf, and surprised to see that by default, max number of smtpd that will be spawned by Postfix is 100.
In normal situation, this won’t be a problem because it’s lightweight. But when there are 100 smtpd processes servicing brute-force attacks of spammer bots, the server will be disturbed.
So I changed this line in /etc/postfix/master.cf :
smtp inet n – – – – smtpd
Into this :
smtp inet n – – – 10 smtpd
Now postfix won’t spawn more than 10 processes at maximum, slowing down the spammer considerably.
Personally, I think old punishment styles such as, oh let’s say “hanged, drawn, quartered” should be reestablished again just for them, spammers.
No, I’m not joking.
OK, ok… but I think that’s the only punishment that would be able to effectively stop people from spamming. So sue me 🙂