Weird Scenes Inside The Gold Mine

... or "Potential Hazards of the System Administration Business" and an appreciation of a classic Unix book (apologies to Jim Morrison and The Doors for ripping off their song title)

One of my prized posessions is my 1999 copy of Kirk Waingrow's "Unix Hints And Hacks". Sure, it's out of date, but as with many things in Unix/Linux, a lot hasn't changed. It inspires me due to its long-range perspective even today. This post is specifically inspired by Chapter 10: System Administration - The Occupation. That's the sort of real-world experience chapter most Unix books lack. The following is solely my own perspective and Waingrow is not responsible for any error!

---

All jobs are not created equal. Choose wisely as you accept Support Engineer, Technical Support, Developer, Support Technician or any such job. Any of them can go south due to personal situations, market conditions or company strategy, but I think I've identified a few types of issues that are common in this field. Watch out for...

THE MAIL DROP AVALANCHE

Use a special email address for job offers. These days, the tendency of businesses to push much HR activity to the web, and the possible tendency of yourself to browse listings rather than contact companies directly will lead to an ornery amount of job search spam. When the listings responses start to share your address (many HR outfits have multiple sites and names, or have been purchased by other outfits) it can get out of control, so make sure the search doesn't gum up the works in your main or personal email account. Listing sites and headhunters haven't performed nearly as well for me as a grapevine network or looking at a company's job listings on their own company site.

THE WALL OF CONFUSION

Use third party research or your grapevine to try to find out what's happening inside a company before you interview. There might have just been some turnover that could not only result in the loss of corporate knowledge but could have confused relationships between job functions and between departments. Like "Responsibility Creep" examined below, you may be seen as fresh meat and as the repository of work offloaded by others. This is possibly the hardest thing to discover about a potential employer. Glassdoor.com and other sites may be useful but that's not a sure thing, since both disgruntled ex-employees and current managers can post comments. Same with Reddit and so many other places. Try any local business journal newspapers/sites for stories about the company or industry.

THE ENTRAPMENT CONSPIRACY

(Note: the next predicament is couched in sneaky, dramatic possibilities but could also result from entirely appropriate reactions to business incidents.)

There exists the possibility of your joining a company at which a legal issue is already in progress. This is not common but considering the state of computer security today, it could increase in frequency. Let's say John Juggler is on the board of a company that relies on internet presence to do business. The company's 24/7 server exposure and intentional lack of a full-time administrator (to save money on employee costs) has resulted in the data loss of 90,000 complete customer records. We all know that this never happens but (in my invented example here) it did and since there could be legal repercussions, not only have they hired an admin but a Computer Information Security Officer was shopped for. You got the job. There is indeed a legitimate need for such a person. There is also a need for Juggler's organization to have a fall guy in case anything requires it. In the board room, sanitized language is used by Juggler and others, as they're looking at the likelihood of future rules in the coming year imposing a 90-day maximum to apply a mandatory data breach notification procedure. The cleanup could go longer than that. Juggler also makes a note to talk to the lawyer about how one might legally prevent a CISO from performing the federally mandated notification for as long as needed to circle the wagons. Good times! A new CISO would benefit by doing a security audit, as soon as he/she had the authority, noting time/date stamps and logging info on and after the CISO's hire date. That way, one could indicate to the Fed (if they ask) when one became aware of the situation. And that new CISO should send out best practices emails to all users immediately upon taking the reins of responsibility, and regularly after that. It is also incumbent upon the new CISO to assess how serious upper management truly is about getting on board with enforcement of new security procedures, which could create friction with those who would protest as to complexity and ease-of-use. Security improvements need to be implemented from the top down. They must come from the board room.

RESPONSIBILITY CREEP

I've seen this happen with remarkable speed. A bunch of people I didn't know in a company with which I'd just started began calling me for everything. When given a task, they instinctively sprung into action by phoning the new guy to do it. Some duties were unrelated to my job description, but I didn't respond with the idiotic "That's not my job" retort. I countered by researching the organization chart, finding out who answered to whom, and got bureaucratic. I indicated that I could get involved if their boss, Person X, talked to my supervisor, Person Y. See, it's my supervisor's (and possibly a client's) choice as to what I do daily. Not anyone else's. So properly one goes through channels in order to change that to-do list. Things began to cool off mysteriously after that point. ["It ain't what they call you; it's what you answer to." - W.C. Fields] Another variant of this can happen if deliberately intended by the company to economize on people: for years I've seen job announcements that almost went on forever, detailing activities that described what it might take two to three people to do. An announcement for Tech Support? Check to see if it involves additional desktop support, on-call availability, website help, DBA work, network management and the ever-present "other duties". This predicament may even be forced onto unsuspecting managers and supervisors if someone above controls the funding for hires, and begins allowing one hire for every two attritioning out. I saw that one happen for years; it wasn't middle management's doing and I suspect was completely contrary to middle management's well-being, wishes and needs.

---

I don't mean to discourage you (variants of the above situations could happen in many industries), but only to help warn against any rude awakening that might be in store. Talking with customers of the organization you're considering can also reveal helpful information not available anywhere else. Good luck!

Layoffs in IT? Not so fast.

I don't know what it is, but I know what it isn't.

For some time there have been news stories such as this one that state that the big layoffs that the infotech world is now seeing would be because of the "move to the cloud" (the customer adopting a rental rather than an ownership model of IT assets). Baloney. Such stories accurately convey what they know of numbers of workers no longer employed, plus estimates of what's to come, and then proceed to prognosticate about the "why" of it all.

If they can, so can I.

I have some experience at a cloud provider which rented not only virtual but physical or co-located assets. Referring to that experience and totally off the cuff, I'll illustrate three categories of asset that give the customer any IT capability:

material assets
structural assets
maintenance assets

The material assets would be the physical hardware. The structural asset is whoever is architecting the nodes and networks together (which determines what to buy or rent). The maintenance assets would be those responsible for operation. The only one of these asset categories affected by "the cloud" (the question of renting or owning) is the first one, the material. We got lots of calls from customers that wanted help on the other two (either operating servers or re-architecting networks) but since we did not sell service, there was little we could do, although we did WHAT little we could for them (restarts, minor investigation). This is my point: they still badly needed help with the second and third asset categories. Some customers tried to get by with no administrator, simply calling a server maintenance company when the box was down and judging a full or part-time administrator to be too expensive. Sometimes we would be surprised to speak with an actual professional administrator and employee to the customer. But you see the situation: whether renting or owning the gear, a business must still get someone to set up the way the computers interact with the world and each other, and must still get someone to perform day-to-day operations; servers don't run themselves any more than cars decide where to drive.

That's why I said "Baloney" at the beginning of this post. "The cloud" is simply an ownership distinction. The only way a cloud situation cutback could affect human administrators would be in a Managed Hosting environment, where you rent not only stuff but people (a wildly expensive prospect). Most servers are rented in the Unmanaged manner (bring your own expertise).

So if the layoffs can't be explained by the change to a rental model, what's up? My guess (I made good grades in Economics) would be that it's the general level of economic activity in a particular country. This seems to be going on worldwide. Although the Information Systems Security Association says that general labor needs between now and 2020 will grow at least 7%, and that IT Security needs will double or triple that in the same period, we still in the short run see cutbacks. I think both observations are true. We do (pretty much) know that for the two Obama presidential terms, GDP has grown less than 3%, which is poor. I say that IT health is directly related to general business activity, since IT is only a delivery device for business activity. If companies are merging, selling out or otherwise closing, there is less immediate need for us, the IT employees. But that changes as business climate improves or degrades.

I don't have any clever math to prove this. Only the logic that since "the cloud" does not give an organization the ability to fire the IT staff, you have to look elsewhere for the layoff impetus. Maybe employee cost goes up because of insurance price hikes. Maybe a competitor is trouncing the company's main product. Maybe the directors of the outfit intended to sell out at this time anyway. Is there a lawsuit from the owners of a similar design? Whatever.

If the general business environment improves, whopee. We get more data so some Eco major can do a thesis. We'll see. But no, rented servers are not the 'mechanical man' supplanting real people.

What Happens When You Ping In Solaris??

You may have been there: you try pinging in Solaris (in my VM running Oracle Solaris 11 specifically) and you find that it's necessary to add a flag, like this:

ping -s (hostname or IP) 5

The flag, I think, means "send packets" and you can specify the number of packets after which to stop at "5". If you don't it pings away until Control-C is invoked.

But what crazy thing happens to get you to that point?

If you try

ping (hostname or IP)

... what happens?

Here's something that hopefully wins you a beer somewhere. The response is this: "ping" does not fail, but returns a simple confirmation. It does not return individual packets at default; that's what the flag is for. It responds to...

ping (hostname or IP)

like so:

(hostname or IP) is alive

You can thank me later.

You Learn Something Every Day

...Every day you look, that is. Today I was doing another Udemy video course and the video was showing the /sbin directory on a popular Linux distro. The usual suspects were there but I noticed, in the several versions of fsck, among stalwarts like fsck.ext2, fsck.ext3, fsck.nfs and so on, one for Minix, of all things, which is indeed still around. But I noticed something even more odd, that I wasn't sure was a mistake, or a smear on my screen: fsck.cramfs.

CramFS?! Is there a filesystem by that name? Surely that's some Intercal-type April Fool joke.

It's no joke. And don't call me Shirley.

The Compressed ROM File System is, like the name suggests, designed to take up very little space. It's often used for embedded systems (the whole thing) but also as an assist for bigger systems. It's really little. A file has to be smaller than 16MB and an entire file system is restricted to about 270MB. Although it's rather old and other schemes have taken over its uses in many Linux distributions, Debian continues to use it for initrd images and I understand that one or another version of SUSE uses it for install images. To maximize smallness (or perhaps stability) it's read-only, although it ships helpfully with some utilities like mkcramfs; that and others are contained in the Util-linux package.

Due to cramfs being obsolete by some arguments, many systems now use the also-weirdly-named squashfs. The tinyness theme continues. Ubuntu, Mint, Debian, Arch and many others currently use it for LiveCDs. Utilities for squashfs are available as well, and mksquashfs and unsquashfs have been ported up to Windows 8.1 at last notice.

And on top of everything else, both cramfs and squashfs, while they exist in a compressed form, do NOT, repeat, do NOT require uncompression to operate. Wow.

So if some admin tries to blindside you with insane Linux terms like these which in an alternate universe would probably not exist, thank me later. I would have lost a bet at the bar after a meeting due to tools named this way.

But first...

Before returning you to our regularly scheduled programming (Linux issues and security complexities), I'd like to do yet another list aimed at what professionals or security advocates seem to need to do on a regular basis with respect to the other 999 out of a thousand people. Imagine, if you will, a person of general knowledge who just got worried about computer security. This person asks you "What should I do now?". I'll give two lists as answers; the first will be for general users and the second will be for small business situations.

WHAT TO DO NOW TO IMPROVE YOUR SECURITY STANCE (for the general user):

1. BACK UP ALL DATA AND APPLICATIONS- Having only one copy of something is risky. Establish a routine and specific location for copying all data on all computers to a storage device or location. An automatic cloud arrangement is better than nothing but that cloud (a cluster or network or rented storage device) is not owned or principally controlled by you. Consider ejecting hard drives or thumb drives that can be placed in a fire protection safe, either on premises or somewhere else.

2. DOCUMENT ALL CONFIGURATIONS AND SETTINGS - All the operating systems and applications that your computers run are probably not at default settings anymore. In documenting all settings, you as well will be inventorying all the applications you use. Network and internet provider information, including contact phone numbers for tech support, would be needed as well.

3. CHANGE ALL PASSWORDS - How many months has it been since you changed pw's on some things - or anything? And never EVER stick with a default password; these are known by the bad guys.

4. UPDATE ALL VERSIONS OF SOFTWARE AND APPLICATIONS - Free security updates are often available for antivirus, operating systems, apps and other aspects of what you do. In the rare case of an update breaking something, backups will be what saves you.

=======================================================

WHAT TO DO NOW TO IMPROVE YOUR SECURITY STANCE (for small business)

1. BACK UP ALL DATA AND APPLICATIONS - Separate CD/DVD copies of all apps and OS's would be good, but saves (dd or other bit-for-bit) should do. Best would be to have a copy of the backup close at hand and another offsite.

2. DOCUMENT ALL CONFIGURATIONS AND SETTINGS - Look into inventory programs for this purpose, or having your staff person write a script. Scripts may already exist that can be customized or chopped down for your situation.

3. CHANGE ALL PASSWORDS - Enable password aging and imposition of use of complexity, as with mixes of capital and lowercase letters, special characters, disallowing dictionary words and password reuse and so on.

4. UPDATE ALL VERSIONS OF SOFTWARE AND APPLICATIONS - Here's where backups are important if any update breaks something. Multiple versions of some language like PHP might be needed for some web page situations, so updating could be tricky; check with your IT person or contract webmaster. The latest PHP could be needed for one thing, while a specific earlier version could be needed for another.

By the way... latest desktop rebuild story

I was given a Windows 10 desktop due to an associate having given up and gotten a new machine after 2 or 3 years. I, king of the museum pieces, had not seen such a recent desktop so was happy to get a chance to check out what was new. This all-in-one 26-inch touchscreen-with-wireless keyboard and mouse resembles to me more of a huge phone than a computer, but I'm not looking a gift horse in the mouth.

Until seeing the user-created files and such still hanging around. So it was time to bring the OS back from the recovery partition (staying with Windows for the moment), at least to see how that would fare. In the future, other OSs like Linux could come in via one of the only two USB ports provided (no ethernet or DVD burner included) but I stayed with what was on board as an experiment.

Lenovo's recovery app behaved as advertised. BUT as will be the case often, it won't go back to Windows 10 if it wasn't already there. This "box" began life at Windows 8. So it replaces that system, which wants about 225 updates in two or three hunks to get up to speed. Now, in about two days, the computer discovered that it could be Windows 8.1, so that was another 137 updates that it needed. Then another 10 security updates after that. Then another nine optionals out of ten (I elected not to put Skype onto this computer).

Then there were the little things. Put an adhesive cover over the camera lens. Grab a few utilities I favor (like SpyBot Anti-Beacon, Spyware Blaster and Mrs. Clinton's IT company's favorite, BleachBit (available for Linux, Windows and on command line for OSX). Uninstall lots of start menu crud. Go over it with screen cleaner. Oh, and if you do something similar like this, give things about a week to settle in. Some security updates won't go in until others do, so let's not assume Redmond got everything in the right order.

No Joke

The future of spamming and other sorts of online skulduggery looks so bright, we'll have to wear shades. The gag observation about toasters having a need for IP addresses (network access) is already losing punch. Over the last few years, we've seen quite a progression. First, remember when this sort of thing began showing up?

Several cable providers have offered over the last few years security systems operated via smartphone or tablet app (and I assume a browser version optimized for a desktop/laptop situation, I don't know). Then shortly thereafter, curious experimenters found holes in the application code, vulnerabilities in the wireless capability as well as the electromechanical controllers addressed by the application and so forth.

Then a year or so later, along comes...

Here's a refrigerator that has cameras inside so you can check current stocks of food (it temporarily turns on the light in there so you can see) from your phone while you're at the grocery store. The door also has a message center, allowing the phones of everybody in the family to send to the screen and leave notes "on the door" about where they are, who needs to be doing what when, etc. I haven't heard about anybody breaking into the embedded processor serving up all these pretty graphics yet - but I haven't had time to find out, since...

Only weeks later, here we have:

This is a new kitchen oven's application that, through your wireless router, has the oven send info to your phone so that you can control the temperature setting and cooking time from anywhere your phone can get a signal. Assuming of course that you're not two floors down in the elevator of an underground parking garage with no repeater antenna in the elevator car.

NOW jump ahead to less than five days ago: I was talking to a security consultant who, I noticed after a minute, had a small bandage on her arm. I asked about the injury - but it was NOT an injury. It was a rejection problem with her implant. An implant for a medical reason? No. It was for holding information that could be scanned... like a business card. She then mentioned the possibilities involving having an implant that could do short-range scans of network activity, or simply be satisfied with passive collection...

I already use a wallet with shielding in it so as to interfere with unwanted (surreptitious) scanning of my credit card information from hidden devices (my buddy got one of his card numbers swiped at a Renaissance Festival this way recently; he did not use a shielded wallet at that time). But I digress.

The point of this musing is that the Internet Of Things is, or soon will be, bigger than any human-operated network. And that this 'thing' network is being peopled by embedded, unmanaged or lightly managed printers, appliances, phones, automotive controllers, software and who knows what else - that might have access to flash storage or even a drive somewhere. Storage + network access sounds like a nice spamming or 'bot command-and-control outpost to me. And the growth of this lightly managed ecosystem is fueled by the convenience demands of people who, to put it mildly, are not network security engineers. Nor would their kids be network security engineers, either, to whom they've given droids and iPhones as toys. From this point it looks easy (since with IPv6 everything can have an address) for lots of chaff activity to overwhelm the network maintenance people, whose staffs are minimized for cost control reasons as it is. And chaff activity is now threatening to make Email unusable, in the same way that usenet was killed by spam posts. But this time it's not just a single application or feature that could be killed - but the whole communications system that could be clogged.

As soon as I've come up with a quick, easy and cheap solution to all this, I'll post again. Gimme a few days. [EDIT: well, whaddaya know. I'd considered joking in this article about wearable Faraday cages that we might need soon, but checking my mail, I'd gotten an ad for this yesterday: The Scott eVest clothing line, now with PAN, or Personal Area Network!. It's already effing here.]

Update of the updated update: According to the Information Systems Security Association (in a recent IoT webinar soon to be documented on YouTube), the IoT numbered around 14 million devices two years ago, and by some projections will hit 50 million by 2020. Several issues on the webinar's agenda prompt me to wonder how one would control what information was being exfiltrated through, say, a home router, and whether or not you'd want that in a hyperconnected future world. Putting aside the worry that every amateur network admin would instantly be committing felonies by running afoul of export restrictions when the cheap electric toothbrush or pacemaker phoned home to China, I speculate that control of data flow and connection requests could be automated with some program resembling antivirus with firewall, operating with an agent at the router. Of course, users have been known in periods of difficulty to disable A/V, firewalls and all else (or to elect not to install such applications at the outset), so perhaps a bit more attention is needed here...