But This Can't Really Happen... much?

Here's the recent story on that deletion mistake, which was a marketing ploy by somebody (HOAX), but it resembled actual incidents I recall from my days working with a hosting provider:

Guy Deletes Entire Company

Yes, it's indeed possible for somebody to come up with a script that has a mass deletion command that isn't properly limited to the current directory or restricted by file privileges (in linux, Microsoft, Unix, Mac, what have you). The Admin or root permission is really powerful, and you should remember to include in the batch file or script the highly specific location(s) that any destructive command needs to work on..

But how would you architect some protections into this? One could go on and on, but the story seems to suggest a one-man operation without a staff, meaning without a professional admin, or even admin company (outsourced). Common situation. Even such an outfit as this one-man-band could benefit from two backup servers (the one in the story had an open connection and presumably its drives were mapped/mounted, therefore got the deletion command along with all other nodes that could be reached). What I'm suggesting is that, with two backup servers, you give one a regular address and the other a private, nonrouteable IP. Script a dd or other copy command from the regular server to the private server and never connect to the private server directly. That makes the first backup server a "staging area" and the second a pseudo-offline location.

Another thing to think about is to make the two backup servers physical (not virtual) servers, since if anything went wrong at those locations (and much of everything else was virtual in order to save money), you could do recovery efforts there since you'd have exclusive access to the drive(s) on those boxes, rather than on some cloud or multiple account arrangement, which would possibly overwrite data with other customer's stuff. I'm sure there are other and possibly better ideas but the simplicity of this second server as a fail-safe is compelling for this more-than-plausible scenario. And further, the backup scheme could treat the first server (staging) as holding only the last one or two backups so they'd be at hand for rebuilds and the second pseudo-offline private server would be for the archiving of a larger number of backups.

[Edit: While the above story about the guy deleting the company with a single code line has its implausibilities, I was told by a colleague about a similar situation having actually occurred a few years back. A local bank had set up much of its capabilities virtually, not just with servers fulfilling specific duties but with virtual networking as well (at least a few virtual switches). Something went wrong in accounting and the cloud provider didn't get paid on time and about 100 boxes just disappeared. That's the way it works when you miss the rent - something else happens automatically (knock at the door, late fee, whatever). In this case, I understand backups went somewhere else (hooray) but the boxes themselves had to be rebuilt and reloaded from backup by poor Mr. Admin - which was one guy, not involved with the problem but still getting the heat. Just another day on duty!]

This is the way it's done.

My congratulations to the maintainers of Linux Mint, the popular Debian-based Linux desktop distro. Two recent posts at their blog, namely

All forums users should change their passwords.

and

Beware of hacked ISOs if you downloaded Linux Mint on February 20th!

are examples of an open source maintenance organization admitting that there have been security issues AND getting word out with clear explanations and instructions for fixes with breakneck speed. Why can't many government and business organizations take heed of examples like this? Good work, guys/girls! This quick work is an example to us all.

Stuff Learned From Current And Previous Jobs

1. In a disaster, always be the first to ask "Where are the backups?". [This presumes that you are NOT assigned to do them. If you ARE assigned to backups, immediately locate the tested and complete backup needed and be ready to transfer copies to all affected locations.]

2. On ambush calls (where you answer the phone and it's four people on speaker - business to business situation - your instant hot seat at the meeting): it's not as bad as you'd think. The call proves that there's a lot of money and credentials at the table that haven't yet figured the issue out. They may bluff or bluster but underneath, they're worried. De-escalate with tone!

3. In an actual emergency, ask yourself (PRIVATELY) whose emergency is it? That's a clue as to where the solution will be.

4. Regardless of what's happening, find out how events are being documented, by whom and where.

5. Regardless of what's happening, find out how others can follow events in real time.

6. Regardless of what's happening, find out how to verify what you're being told.

7. Find out how many others (admins, users, owners, customers) have access to the work area, and if any of the above need clear and fast warning in case of application restarts, system downtime, reconfigurations, file location changes, etc.

8. Answer this: do you have permission to do the work, and how do you prove that?

9. Be diplomatic when discussing technical documentation; you might be talking to the poor soul that had to write it. Even sketchy documentation was probably very hard to obtain.

Another Clever Stall, putting off the Security Onion post yet again!

Important and weighty observations:

1. I have not (in at least 15 years) seen Microsoft Malicious Software Removal Tool find anything. No doubt this is a tribute to my top-notch adminning - or that I don't use Windows enough.

2. I see many suspicious listings for IT positions, and have for years. I wonder about, say, ones for Linux System Administrator, yet into the second paragraph creeps things like Active Directory. And it's just as unfair for that Windows Administrator offer to eventually mention that little Linux project... [translation: we need two or three people, but we'll see how far we can get the budget to go...] And it gets worse. Did you hear about the hospital that paid the ransom to get back file access this week? News stories are mentioning that healthcare companies don't spend a lot on security these days, as if we needed to be told. Looks like the two issues (security problems and responsibility creep) have the same cause: budget problems.

3. This must be a good time to be a spammer. Need a server from which to do the job? Half the servers on the air probably don't have a dedicated admin due to cost-cutting (or some outfits never having had an admin at all). There are many server/website "admins" that haven't logged into their boxes in a year. Who's watching password aging? Patching? Upgrading? Is the box even up?

4. If you need instruction, Udemy is having a sale last I heard. And they do this fairly often: many pricey courses are offered for short periods at only $10. I'm taking a few now; instructors are hit/miss but all have at least been worth the time spent.

5. And in addition to Security Onion, I'm looking at DEFT (Digital Evidence and Forensics Toolkit).

True Life Adventures In Server Support

A customer opened an issue with us, saying that his server emailed him a crash report. It did, and it was included in the incoming ticket. It was quite detailed and elegant in a way, well designed and prepared... and we had no earthly idea what it meant. The supervisor didn't know. The advanced support people didn't know. For awhile, the internet didn't – until I'd searched enough random lines in the report to come up with some hits on abrtd, and later it made sense due to the customer running CentOS. We though at first that it was a system crash from some 'abort daemon' but no, the name was an acronym: Automatic Bug Reporting Tool (daemon), a Red Hat/CentOS thing. The web hits further suggested that this tool watched for application crashes, not system crashes. We figured that uptime results would confirm that the box hadn't gone down, but the customer spent another two or three tickets finding the server password...

When the password was provided in the third ticket (I'd been the only one picking these up since I'd gotten the first one and was perversely curious), I got in and uptime confirmed that the box had been up for over 50 days. Then, I checked around the logs for something to correspond with the cryptic reports from abrtd. No luck. Fortunately, the net hits mentioned cli commands for this daemon, one of which was “list”. This was the jackpot, or at least the only promising lead. It gave six incidents (little report notices of about 5 lines each) with a timestamp and the app concerned in the fail. I recognized 5 out of six as normal for a CentOS box with cPanel, but #6 was odd: something or other ending in .py, and sitting in somebody's home directory.

Even a novice would suspect something amiss. I sprung into action and nicely suggested that perhaps he'd paid a developer to write this and might indicate to us what the script did, if anything. Also, the tool would probably flag as a problem some application activity that it hadn't been told about.

Success. The customer speculated that it was a local tool and that he'd look into it. Ticket retired.

The moral of the story is that these modern operating systems are so extensively appointed that you can be blindsided by features that no one knew about – even in a roomful of professionals. I am continually amazed at what I learn just by either answering a ticket, answering the phone or just standing around.

I Apologize In Advance For This Post (I swear the SecurityOnion test is coming!)

One should not have to think about this one, or write about it. I see server operations "professionals" almost five days a week that can't be bothered to lift a finger about security until after it's too late... but I've just seen something that's worse, if you can imagine that.

It appears that another major social media site has been hacked, but this one (which I won't dignify with a mention) is one devoted to the furthering of cheating. According to media reports it bills itself as “the world’s leading married dating service for discreet encounters”. In a world in which the internet is becoming more of a party line every day, the idea of establishing a central point for illicit activity would seem not to be a very good idea. But some evidently avail themselves of this.

Now, the guys I was mentioning before (Admins who slack off on security) are sometimes people who are intimidated by such simplicities as Windows Update. But simply being nontechnical doesn't mean being stupid. Often the uninitiated hire somebody else to handle that stuff. Fine and dandy. Some of them put it off until a problem impels them to hire outside help. Fine and dandy. That's one level of the problem. That's not as dumb as cheating on a spouse through a maybe-insecure site that publicly advertises its purpose as enabling illicit hookups.

This has implications for the security and server operations industries. How do you protect against a user base that pushes back frontiers in stupidity? This creates an internal threat that is dynamic (apparently getting worse by the year). Talk about a cost of doing business. This is the end of the post. I have no idea how to fix stupid (as the comedian Ron White says). I guess that making sure to do backups is the only answer, since by my logic, sooner or later (depending on how many bad hires one makes), you're gonna be rebuilding something.

Two Items Actually Connected With Linux and/or Security And Technology!

First, let's get the Windows matter out of the way: I noticed something on one of my Win7 boxes that will be relevant to the security updates Redmond just pushed out. I'd seen McAfee VirusScan interfere with the installation of several KB items (updates) before (to fix it you just set the agent to STOP, do all the download/install/config stuff then set it back with START), but just experienced the same problem where McAfee wasn't on board. I'm trying out Bitdefender and realized that it has a comprehensive status control panel, on which it checks for, not only its own definition files but for things like Adobe updates and Windows Updates. In this case, when I checked for updates through Bitdefender's panel rather than through the ordinary OS updater (WU panel under Security in W7), it worked ok. Just sayin'.

Second, here's the page for a nice security-based distro some of you already know about:

Security Onion

It's based on the easy-to-use Ubuntu and includes tools like Snort, Snorby, Network Miner and Bro. I'll do a more in-depth analysis on it as soon as I kick the tires.