I spent quite a bit of time yesterday mitigating an ongoing attack on one of the PN&R servers. I was browsing the site logs as is my habit with a coffee in the morning and I saw hundreds of emails backed up in the postfix queue, all with GMAil addresses.
My first thought was. “Here we go again, GMail has tweaked some setting and all my email is backed up.” To be fair, GMail implements their email system with very strict adherence to the protocol, and if they say something is wrong with your email, you probably should see what it is. Their stance is that you have to prove that your email is worth putting in their customer’s inbox.
I took a second look at the postfix logs and quickly realized what was going on. I’m not going to describe the attack in detail, but it is one that I lectured on when I was teaching network security. Even knowing the attack mechanism, it took all of the morning to stop the attack, fix the hole, check the systems, and turn everything back on.
The result is that the PN&R systems are that much more secure, so I consider it time well spent.
Right? I’ve never understood the motivation for attacking random servers, or for the nutjobs who QRM POTA ops. What is wrong with these people? Clearly something.
I saw in the logs that email from us to Hotmail addresses aren’t being delivered, it’s a result of the earlier attack. GMail was bouncing them, but Microsoft is much looser about protocols and would have accepted and then seen them as spam, which they were. Nothing to do from the PN&R side, over the next few days Hotmail will start allowing our emails back in.
If you happen to be getting forum digests delivered to a Hotmail address, you might miss a week.
The site’s been down for a few weeks, just after I left on an extended camping trip in the midwest we were hit by a serious denial-of-service attack. I managed to pull the site offline but didn’t have a chance to see what the damage was until I returned a few days ago. Fortunately it really was a DoS and there was no data corruption. We’re back online with a few new defenses.