Sunday, March 23, 2014

I was only a bystander but here's what I saw...

So this guy says he's getting a bunch of unsolicited SMTP interrogations and it's filling up his logs and requiring him to check in a lot to make sure he's still got room in that directory so as not to strangle commands and crash the server. Could we "fix it"? Actually, no, since we don't manage his stuff, but we checked into it for him by asking another department with the clearance to see more that we can. "This guy" fears some kind of DDOS but our superiors point out that the incoming SMTP connection attempts aren't in amounts any greater that 200kb per second, and add to that the fact that the server operator admits that email isn't that active. I get onto the command line at one point and watch uptime, top, df and so on, noticing not much load or activity.

Apparently there's not much to any of this... except the fact that the guy's logs have been filling up and this effect took the server down at least once. The SMTP requests are coming from many obviously bogus IP's and not actually generating mail. BUT the connections must be logged, so /var/whatever is overloaded at some point and Crash. This looks to my inexperienced eyes like an unusual form of Denial Of Service.

A few of us came up with a strategy for the server operator. First, check in often if nothing else. Second, perhaps come up with a cron job or two that will archive or dump the affected logs based on size. Third, if he has some kind of web hosting control panel like Plesk, there's probably an app that's a front end to crontab or logrotate that would easily schedule size-based dumps as above.

And it came to a head on a weekend. That's the main reason we bought into the DDOS conspiracy theory (much skullduggery happens from Friday to Sunday), but the guy probably needed to set up rotation anyway. Serves him right, as well as us all. Also, he could be misconfigured in 15 other places, in addition to Mr. BogusIPs coming at him.

The moral of the story might be that any protocol activity on any server or application THAT MUST BE LOGGED can constitute a kind of vulnerability, since if not checked, the log directories involved would fill up at some point. And if you didn't throw bogus requests at your target in multi-gigabit-per-second amounts, you fly under the radar of either the server or the network outfits. It's an effective trick, until the target figures it out - or unless the target configured logrotate effectively in the first place.

EDIT: I should know better than to gravitate to conspiracy, particularly when I recall the quote "never attribute to deviousness what can be adequately explained by incompetence"... Another explanation for the above facts (pointed out by someone with more experience in these matters than me) is that, in configuring some DNS matter for email purposes, maybe somebody got one digit of an IP address wrong... and our "victim" is getting someone else's mail through no fault of his own. Think about it: 175 kbps or so, random IP's... yeah, maybe so. It would still be a clever ploy to deny service due to reboot inducement, but again, if one does logrotate right, the issue never comes up.