DRaaS

Software as a service, Platform as a service, Infrastructure as a service… might as well list Accounting as a service, HR as a service, Security as a service, Programming & Development as a service, Service as a service (actually that last isn’t a joke, as Customer Support has been offshored for many organizations now)… can every business process be outsourced or commoditized? I don’t know but I know that I haven’t seen much new – or old – in the service economy that compelled me to conclude that the benefits outweighed the hazards of paying outside sources for everything. But there may be a category now that makes more sense than many: Disaster Recovery as a Service.

Cloud providers can give you a rented virtual (and often the additional choice of rented physical) infrastructure. You can rent online storage for backups. You can dispense with information technology staffing by hiring server admin companies on an as-needed basis (I don’t recommend that but it’s a widespread practice). You can hire security experts as needed, hopefully before they’re needed. But what if there was a way to combine solutions for many of these needs into one strategy? As I understand it, Disaster Recovery as a Service uses a massive backup as a fix for not just hardware failure or power loss, but for many things this side of ddos mitigation (or other problems outside your network). One particular company with a booth I saw at a recent security conference appeared to work like Carbonite backup, except for your whole network inventory. All nodes (desktops, laptops, workstations, servers) plus all network devices like routers and switches are “cloned” at the provider’s data center. This idea stores not just data backups but whole operating system builds, with users, desktop environments, configurations, applications and stray documents. And in the network devices, all the addresses, routings, rules, configs and system versions are included. These are (in this particular company’s instance) not just snapshots restricted to one point in time, but constantly updated “clones” of All The Things.

A “Carbonite for your whole shop” approach would, as described here, seem to rely on lots of bandwidth to do all the constant updating (plus think cost for that in addition to network capability matters). Also, the securing of the connection to the provider location or cloud introduces an issue. The defense stance of the provider data center itself would constitute another. After all, grabbing backups has already proven profitable to criminals worldwide, who didn’t have to frontally attack a victim’s main arrangements. But there’s also promise in the approach. When your hardware and connectivity become ready after an event, you could come back from fire, flood, tornado, hurricane, earthquake, physical strikes like an eighteen-wheeler smashing through your office wall, node breakages induced by Windows Update, System Update or the like, worm damage, maliciously encrypted data blackmail and most problems related to geographical location (sinkholes?).

I can see potential challenges here with cost, security and network capacity. But if these can be handled then DRaaS is a new approach that could overtake legacy methods of the old offsite backup, reliable as it is. I’ll be watching the market for this product class with great interest.

Weird Scenes Inside The Gold Mine

... or "Potential Hazards of the System Administration Business" and an appreciation of a classic Unix book (apologies to Jim Morrison and The Doors for ripping off their song title)

One of my prized posessions is my 1999 copy of Kirk Waingrow's "Unix Hints And Hacks". Sure, it's out of date, but as with many things in Unix/Linux, a lot hasn't changed. It inspires me due to its long-range perspective even today. This post is specifically inspired by Chapter 10: System Administration - The Occupation. That's the sort of real-world experience chapter most Unix books lack. The following is solely my own perspective and Waingrow is not responsible for any error!

---

All jobs are not created equal. Choose wisely as you accept Support Engineer, Technical Support, Developer, Support Technician or any such job. Any of them can go south due to personal situations, market conditions or company strategy, but I think I've identified a few types of issues that are common in this field. Watch out for...

THE MAIL DROP AVALANCHE

Use a special email address for job offers. These days, the tendency of businesses to push much HR activity to the web, and the possible tendency of yourself to browse listings rather than contact companies directly will lead to an ornery amount of job search spam. When the listings responses start to share your address (many HR outfits have multiple sites and names, or have been purchased by other outfits) it can get out of control, so make sure the search doesn't gum up the works in your main or personal email account. Listing sites and headhunters haven't performed nearly as well for me as a grapevine network or looking at a company's job listings on their own company site.

THE WALL OF CONFUSION

Use third party research or your grapevine to try to find out what's happening inside a company before you interview. There might have just been some turnover that could not only result in the loss of corporate knowledge but could have confused relationships between job functions and between departments. Like "Responsibility Creep" examined below, you may be seen as fresh meat and as the repository of work offloaded by others. This is possibly the hardest thing to discover about a potential employer. Glassdoor.com and other sites may be useful but that's not a sure thing, since both disgruntled ex-employees and current managers can post comments. Same with Reddit and so many other places. Try any local business journal newspapers/sites for stories about the company or industry.

THE ENTRAPMENT CONSPIRACY

(Note: the next predicament is couched in sneaky, dramatic possibilities but could also result from entirely appropriate reactions to business incidents.)

There exists the possibility of your joining a company at which a legal issue is already in progress. This is not common but considering the state of computer security today, it could increase in frequency. Let's say John Juggler is on the board of a company that relies on internet presence to do business. The company's 24/7 server exposure and intentional lack of a full-time administrator (to save money on employee costs) has resulted in the data loss of 90,000 complete customer records. We all know that this never happens but (in my invented example here) it did and since there could be legal repercussions, not only have they hired an admin but a Computer Information Security Officer was shopped for. You got the job. There is indeed a legitimate need for such a person. There is also a need for Juggler's organization to have a fall guy in case anything requires it. In the board room, sanitized language is used by Juggler and others, as they're looking at the likelihood of future rules in the coming year imposing a 90-day maximum to apply a mandatory data breach notification procedure. The cleanup could go longer than that. Juggler also makes a note to talk to the lawyer about how one might legally prevent a CISO from performing the federally mandated notification for as long as needed to circle the wagons. Good times! A new CISO would benefit by doing a security audit, as soon as he/she had the authority, noting time/date stamps and logging info on and after the CISO's hire date. That way, one could indicate to the Fed (if they ask) when one became aware of the situation. And that new CISO should send out best practices emails to all users immediately upon taking the reins of responsibility, and regularly after that. It is also incumbent upon the new CISO to assess how serious upper management truly is about getting on board with enforcement of new security procedures, which could create friction with those who would protest as to complexity and ease-of-use. Security improvements need to be implemented from the top down. They must come from the board room.

RESPONSIBILITY CREEP

I've seen this happen with remarkable speed. A bunch of people I didn't know in a company with which I'd just started began calling me for everything. When given a task, they instinctively sprung into action by phoning the new guy to do it. Some duties were unrelated to my job description, but I didn't respond with the idiotic "That's not my job" retort. I countered by researching the organization chart, finding out who answered to whom, and got bureaucratic. I indicated that I could get involved if their boss, Person X, talked to my supervisor, Person Y. See, it's my supervisor's (and possibly a client's) choice as to what I do daily. Not anyone else's. So properly one goes through channels in order to change that to-do list. Things began to cool off mysteriously after that point. ["It ain't what they call you; it's what you answer to." - W.C. Fields] Another variant of this can happen if deliberately intended by the company to economize on people: for years I've seen job announcements that almost went on forever, detailing activities that described what it might take two to three people to do. An announcement for Tech Support? Check to see if it involves additional desktop support, on-call availability, website help, DBA work, network management and the ever-present "other duties". This predicament may even be forced onto unsuspecting managers and supervisors if someone above controls the funding for hires, and begins allowing one hire for every two attritioning out. I saw that one happen for years; it wasn't middle management's doing and I suspect was completely contrary to middle management's well-being, wishes and needs.

---

I don't mean to discourage you (variants of the above situations could happen in many industries), but only to help warn against any rude awakening that might be in store. Talking with customers of the organization you're considering can also reveal helpful information not available anywhere else. Good luck!

Layoffs in IT? Not so fast.

I don't know what it is, but I know what it isn't.

For some time there have been news stories such as this one that state that the big layoffs that the infotech world is now seeing would be because of the "move to the cloud" (the customer adopting a rental rather than an ownership model of IT assets). Baloney. Such stories accurately convey what they know of numbers of workers no longer employed, plus estimates of what's to come, and then proceed to prognosticate about the "why" of it all.

If they can, so can I.

I have some experience at a cloud provider which rented not only virtual but physical or co-located assets. Referring to that experience and totally off the cuff, I'll illustrate three categories of asset that give the customer any IT capability:

material assets
structural assets
maintenance assets

The material assets would be the physical hardware. The structural asset is whoever is architecting the nodes and networks together (which determines what to buy or rent). The maintenance assets would be those responsible for operation. The only one of these asset categories affected by "the cloud" (the question of renting or owning) is the first one, the material. We got lots of calls from customers that wanted help on the other two (either operating servers or re-architecting networks) but since we did not sell service, there was little we could do, although we did WHAT little we could for them (restarts, minor investigation). This is my point: they still badly needed help with the second and third asset categories. Some customers tried to get by with no administrator, simply calling a server maintenance company when the box was down and judging a full or part-time administrator to be too expensive. Sometimes we would be surprised to speak with an actual professional administrator and employee to the customer. But you see the situation: whether renting or owning the gear, a business must still get someone to set up the way the computers interact with the world and each other, and must still get someone to perform day-to-day operations; servers don't run themselves any more than cars decide where to drive.

That's why I said "Baloney" at the beginning of this post. "The cloud" is simply an ownership distinction. The only way a cloud situation cutback could affect human administrators would be in a Managed Hosting environment, where you rent not only stuff but people (a wildly expensive prospect). Most servers are rented in the Unmanaged manner (bring your own expertise).

So if the layoffs can't be explained by the change to a rental model, what's up? My guess (I made good grades in Economics) would be that it's the general level of economic activity in a particular country. This seems to be going on worldwide. Although the Information Systems Security Association says that general labor needs between now and 2020 will grow at least 7%, and that IT Security needs will double or triple that in the same period, we still in the short run see cutbacks. I think both observations are true. We do (pretty much) know that for the two Obama presidential terms, GDP has grown less than 3%, which is poor. I say that IT health is directly related to general business activity, since IT is only a delivery device for business activity. If companies are merging, selling out or otherwise closing, there is less immediate need for us, the IT employees. But that changes as business climate improves or degrades.

I don't have any clever math to prove this. Only the logic that since "the cloud" does not give an organization the ability to fire the IT staff, you have to look elsewhere for the layoff impetus. Maybe employee cost goes up because of insurance price hikes. Maybe a competitor is trouncing the company's main product. Maybe the directors of the outfit intended to sell out at this time anyway. Is there a lawsuit from the owners of a similar design? Whatever.

If the general business environment improves, whopee. We get more data so some Eco major can do a thesis. We'll see. But no, rented servers are not the 'mechanical man' supplanting real people.

What Happens When You Ping In Solaris??

You may have been there: you try pinging in Solaris (in my VM running Oracle Solaris 11 specifically) and you find that it's necessary to add a flag, like this:

ping -s (hostname or IP) 5

The flag, I think, means "send packets" and you can specify the number of packets after which to stop at "5". If you don't it pings away until Control-C is invoked.

But what crazy thing happens to get you to that point?

If you try

ping (hostname or IP)

... what happens?

Here's something that hopefully wins you a beer somewhere. The response is this: "ping" does not fail, but returns a simple confirmation. It does not return individual packets at default; that's what the flag is for. It responds to...

ping (hostname or IP)

like so:

(hostname or IP) is alive

You can thank me later.

You Learn Something Every Day

...Every day you look, that is. Today I was doing another Udemy video course and the video was showing the /sbin directory on a popular Linux distro. The usual suspects were there but I noticed, in the several versions of fsck, among stalwarts like fsck.ext2, fsck.ext3, fsck.nfs and so on, one for Minix, of all things, which is indeed still around. But I noticed something even more odd, that I wasn't sure was a mistake, or a smear on my screen: fsck.cramfs.

CramFS?! Is there a filesystem by that name? Surely that's some Intercal-type April Fool joke.

It's no joke. And don't call me Shirley.

The Compressed ROM File System is, like the name suggests, designed to take up very little space. It's often used for embedded systems (the whole thing) but also as an assist for bigger systems. It's really little. A file has to be smaller than 16MB and an entire file system is restricted to about 270MB. Although it's rather old and other schemes have taken over its uses in many Linux distributions, Debian continues to use it for initrd images and I understand that one or another version of SUSE uses it for install images. To maximize smallness (or perhaps stability) it's read-only, although it ships helpfully with some utilities like mkcramfs; that and others are contained in the Util-linux package.

Due to cramfs being obsolete by some arguments, many systems now use the also-weirdly-named squashfs. The tinyness theme continues. Ubuntu, Mint, Debian, Arch and many others currently use it for LiveCDs. Utilities for squashfs are available as well, and mksquashfs and unsquashfs have been ported up to Windows 8.1 at last notice.

And on top of everything else, both cramfs and squashfs, while they exist in a compressed form, do NOT, repeat, do NOT require uncompression to operate. Wow.

So if some admin tries to blindside you with insane Linux terms like these which in an alternate universe would probably not exist, thank me later. I would have lost a bet at the bar after a meeting due to tools named this way.

But first...

Before returning you to our regularly scheduled programming (Linux issues and security complexities), I'd like to do yet another list aimed at what professionals or security advocates seem to need to do on a regular basis with respect to the other 999 out of a thousand people. Imagine, if you will, a person of general knowledge who just got worried about computer security. This person asks you "What should I do now?". I'll give two lists as answers; the first will be for general users and the second will be for small business situations.

WHAT TO DO NOW TO IMPROVE YOUR SECURITY STANCE (for the general user):

1. BACK UP ALL DATA AND APPLICATIONS- Having only one copy of something is risky. Establish a routine and specific location for copying all data on all computers to a storage device or location. An automatic cloud arrangement is better than nothing but that cloud (a cluster or network or rented storage device) is not owned or principally controlled by you. Consider ejecting hard drives or thumb drives that can be placed in a fire protection safe, either on premises or somewhere else.

2. DOCUMENT ALL CONFIGURATIONS AND SETTINGS - All the operating systems and applications that your computers run are probably not at default settings anymore. In documenting all settings, you as well will be inventorying all the applications you use. Network and internet provider information, including contact phone numbers for tech support, would be needed as well.

3. CHANGE ALL PASSWORDS - How many months has it been since you changed pw's on some things - or anything? And never EVER stick with a default password; these are known by the bad guys.

4. UPDATE ALL VERSIONS OF SOFTWARE AND APPLICATIONS - Free security updates are often available for antivirus, operating systems, apps and other aspects of what you do. In the rare case of an update breaking something, backups will be what saves you.

=======================================================

WHAT TO DO NOW TO IMPROVE YOUR SECURITY STANCE (for small business)

1. BACK UP ALL DATA AND APPLICATIONS - Separate CD/DVD copies of all apps and OS's would be good, but saves (dd or other bit-for-bit) should do. Best would be to have a copy of the backup close at hand and another offsite.

2. DOCUMENT ALL CONFIGURATIONS AND SETTINGS - Look into inventory programs for this purpose, or having your staff person write a script. Scripts may already exist that can be customized or chopped down for your situation.

3. CHANGE ALL PASSWORDS - Enable password aging and imposition of use of complexity, as with mixes of capital and lowercase letters, special characters, disallowing dictionary words and password reuse and so on.

4. UPDATE ALL VERSIONS OF SOFTWARE AND APPLICATIONS - Here's where backups are important if any update breaks something. Multiple versions of some language like PHP might be needed for some web page situations, so updating could be tricky; check with your IT person or contract webmaster. The latest PHP could be needed for one thing, while a specific earlier version could be needed for another.

By the way... latest desktop rebuild story

I was given a Windows 10 desktop due to an associate having given up and gotten a new machine after 2 or 3 years. I, king of the museum pieces, had not seen such a recent desktop so was happy to get a chance to check out what was new. This all-in-one 26-inch touchscreen-with-wireless keyboard and mouse resembles to me more of a huge phone than a computer, but I'm not looking a gift horse in the mouth.

Until seeing the user-created files and such still hanging around. So it was time to bring the OS back from the recovery partition (staying with Windows for the moment), at least to see how that would fare. In the future, other OSs like Linux could come in via one of the only two USB ports provided (no ethernet or DVD burner included) but I stayed with what was on board as an experiment.

Lenovo's recovery app behaved as advertised. BUT as will be the case often, it won't go back to Windows 10 if it wasn't already there. This "box" began life at Windows 8. So it replaces that system, which wants about 225 updates in two or three hunks to get up to speed. Now, in about two days, the computer discovered that it could be Windows 8.1, so that was another 137 updates that it needed. Then another 10 security updates after that. Then another nine optionals out of ten (I elected not to put Skype onto this computer).

Then there were the little things. Put an adhesive cover over the camera lens. Grab a few utilities I favor (like SpyBot Anti-Beacon, Spyware Blaster and Mrs. Clinton's IT company's favorite, BleachBit (available for Linux, Windows and on command line for OSX). Uninstall lots of start menu crud. Go over it with screen cleaner. Oh, and if you do something similar like this, give things about a week to settle in. Some security updates won't go in until others do, so let's not assume Redmond got everything in the right order.