Givin' Mr. Gregg Some Love

Around six months ago a user came into one of the meetings needing assistance setting up penetration testing capability. Briefly it involved setting up some target virtual machines in a virtually networked arrangement, and as I understood it, one could simply include a VM of Kali linux or Fedora Security Lab or some such testing distribution to get a nice tool set onboard with which to look at the other "boxes", and that would complete the arrangement. The virtualization scheme (a closed-source product) had become a bit problematic in a few ways and thoughts on that predicament have stayed with me. Looking back, I see three approaches that would profitably address such a set of issues:

1. Use the free tech support included with the virtualization product to address the specific networking issues we looked at in the meeting. We're an open source club and although it's possible that we have members familiar with almost any technology, the product was from a company with which our experience was limited. Unless I and others were missing something, it seems to me that a virtualization product (Virtual Box, VM Ware's latest chopdown version, Parallels, Hyper-V, whatever) should provide a basic network out of the box. Just add an assortment of OS's plus one pentesting specialist linux distro and you're done.

2. Guaranteed: most people will hate this suggestion with a murderous passion, but if it were me, I'd eliminate the virtual layer and simply get a switch and three or four separate desktops and/or laptops. For real; no VMs. I'd hook up a Windows box, attach a Mac, add one with OpenSUSE or any general purpose distro in the top 20 over at Distrowatch (right column under the ad block for latest rankings), and the last box would run Kali. Along with a nice Network+ class over at udemy.com, this would be what I'd think would teach network penetration best. After all, building a network is what you'd need to do by yourself first, in order to understand what you propose to penetrate. To keep the electricity bill down, I wouldn't run everything at the same time. I also benefit from not currently being married, so that would get me around the exclamatory reactions surrounding all the wiring and equipment laying everywhere.

3. I only wish, when the user's question originally arose, that I'd had a particular book to hold up in my non-virtual hand: Michael Gregg's "The Network Security Test Lab: A Step By Step Guide". It goes from the physical arrangement (either real or virtual) in the vein mentioned above to dozens of additional topics that flow naturally from the reasons one builds such a capability in the first place. He understands that pen testing is not an isolated capability. One needs an initial and intimate familiarity with the real estate that one proposes to explore.

This review is a preliminary one; it will provoke further articles from me. The book constitutes a syllabus for a class that universities should be giving if they aren't already. Of eleven chapters, 'constructing the lab' is confined to Chapter One. Further chapters address the whys and hows of using what Chapter One allows you to assemble. Passive Information Gathering is given its own chapter, discussing methods from banner grabbing to dumpster diving, in order to drive home the point that all information is not on the network - or perhaps ANY network, and the point that the more noninvasive the surveillance, the better. At least at first. Results from passive gathering will tell a tester how exposed the client is, which will make important reading in the results report. Properly, network traffic analysis and system identification are given separate chapters and the analysis chapter comes first, as the questions will occur on the way in. In the traffic chapter (chapter 3) you'll find Gregg is big on Wireshark as a main tool, wireless or wired as the problem may be. He quickly proceeds from packet basics to real-world examples involving tricks like VLAN hopping and different types of LAN taps. The System Detection and Analysis chapter (4) starts with a hex refresher and proceeds to discussions of services that different OS's tend to implement differently (this is the way nmap and other tools try to ID boxes on the fly). A basic example is the TTL: Linux time-to-live for packets is 68, Windows' is 128 and many hardware devices put it out to 255 (unrestricted). But later, System Enumeration is given its own separate chapter (5). There are further ones for Encryption and Tunneling (6), Automated Tools (7), security problems peculiar to wireless networking (8), malware (9), Intrusion Detection and more malware analysis from a post-intrusion perspective (10) and Forensics (11).

Some high points of those last chapters include discussions of password tools like PassTheHash (yuk yuk) and crackers that take slang like Klingon into account (chapter 6), assessment tool scriptability as with Nessus (NASL), metasploit (ruby or perl), nmap (lua/NSE) and point/click tools like BeEF and Core Impact (chapter 7), and resources on how various entities like retailers look for your phone in order to track you within their stores (chapter 8). I should note that Chapter 9 and 10 overlap; 9 is the introductory level of Malware discussion (using tools like Rootkit Hunter, virustotal.com, etc.) and 10 gets into the heavier stuff like IDS tuning (he prefers Snort since their maintenance site has many preconfigured rules and signatures to get you started).

My compliments to the chef. This is a well-written and current resource that I'll be turning to again and again (I require much repetition to learn and Gregg repeats what I need repeated). He also writes for the Huffington Post website; check him out there.

Review: The Hacker Playbook 2: A Practical Guide To Penetration Testing
by Peter Kim
1st Edition, July 2015 / SecurePlanet LLC / ISBN-13: 9781512214567

I got a copy of this text from a class I just completed, which I'll describe in greater detail when you see me in person at the meeting. What it's important to get down in this review is some descriptions of the content (which is up to date) and that those descriptions be delivered in the proper order. For some reason this first printing either didn't get enough proofreading or the setup editor screwed up, not bolding or increasing font size on chapter headings, nor even numbering them. So I laboriously found all the chapter headings and will organize my comments by them below, for your convenience if you acquire a copy of Mr. Kim's excellent book (after figuring the headings out, I just marked them on their pages and on the Contents page with a highlighter so that they became useful). Also, the descriptive conceit of the book uses analogies to game plays and strategies in American football so as to separate concepts, so keep that in mind. By the way, the "2" designation in the title doesn't mean that you need to look up the previous one; it denotes a second edition which includes all material still useful from Kim's earlier volume, with updates and additional coverage of tools and strategy as relevant to about 10 months ago.

"Pregame - The Setup" amounts to Chapter 1. This covers the physical setup needed for pen testing. He describes your lab as needing such features as virtualization software, particular VMs representing popular OS's, and mentions particular aspects of popular systems such as Powershell in Windows. Finally a "Learning" section has some early discussion of tools and strategies like Metasploitable and issues like lack of secure form in binaries.

"Before The Snap - Scanning The Network" amounts to Chapter 2. Passive observation and discovery is the first topic, noting tools like Recon-NG, 'discover' scripts and Spiderfoot. I should mention that all versions are referring to Kali Linux inclusions. Kali Linux is one of the largest and most popular security distributions and it's referred to so much here that an interesting secondary use of Mr. Kim's book would be as a Kali manual.

IN FACT, Kali Linux is so badass, go get a copy right now if you don't have it, at this convenient location.

Coverage proceeds to password lists, looked at with Wordhound and Brutescrape, then with active tools like Masscan and Sparta. Then some vulnerability tools are introduced like Rapid7/Nexpose, Teneble Nessus and Openvas. Web apps are looked at with OWASP Zap proxy, which is available for Windows, Linux and OSX. Finally, nmap, Burp and straight Nessus are mentioned.

"The Drive - Exploiting Scanner Findings" amounts to Chapter 3. It starts with a Metasploit Framework example which, while comprehensive, assumes previous experience with Metasploit and gives links as to where to get up to speed with that. Then there's a discussion of printers, Nosqlmap and Elastic Search. Then some recent issues like Hearbleed and Shellshock are described (no doubt some stalwart professionals are still vulnerable to those).

"The Throw - Manual Web App Findings" amounts to Chapter 4. There's a general intro to web app pen testing and SQL injections as such, then a generous 15 pages on manual SQL injection methods, followed by 5 pages on Cross Site Scripting. Other topics covered are Cross Site Request Forgery, tokens and fuzzing. Then Kim gives a mention of the Top Ten vulnerability list maintained by OWASP (I'll give it here for your convenience):

OWASP Top Ten cheat sheet

"The Lateral Pass - Moving Through The Network" amounts to Chapter 5. Responder is the first tool mentioned; included in Kali, responder.py looks for multicast name resolution and NetBios information, and uses the Microsoft WPAD vulnerability (there's a TechNet article on that; evidently the service's PAC file points to a config file that's wide open, if you can find it). Then ARP poisoning is discussed from the standpoint of two Kali-included tools (ettercap and Backdoor Factory proxy) and Cain And Abel for Windows. Two methods for getting network access at this point are given, with specific steps. The methods are either 1)with any credentials or 2)with Local Admin or Domain Admin account info. Two tools for manipulating the domain controller then mentioned are SMBexec and PSExec_ndtsgrab, both in their Kali inclusion versions. The convenient strategy of creating access "persistence" is then brought up, listing popular tools for this as Golden Ticket (a Kerberos crack), Skeleton Key (a domain admin backdoor) and Sticky Keys (a sort of automation of hitting Shift five times on a Windows host; this idea uses registry settings, which seems nonstealthy to me if the registry is locked or monitored in some way but this is one of Kim's favorite tools so I'll defer to his experience).

"The Screen - Social Engineering" amounts to Chapter 5. The expected phishing and wireless methods are covered, but there's also the tale of the author's actual purchase of similar domains as targets to take advantage of typographical errors when users enter domain addresses. He links to his complete research paper but also describes this very nasty idea, popularly referred to as "doppelganger domains". If an outfit uses their subdomain for email and somebody mistypes, BANG. They go to the bad guy's location instead. To make a long story short, these similar addresses can be incorportated into links in emails or into icons on which to click. Do you want to maim somebody all of a sudden? I don't blame you. Then there's the methods that involve the risk of physical access, like planting rogue access points, dropping USB sticks with hidden file infectors in the hall or parking lot, purchased devices for smart card cracking, KonBoot on USB for at-the-host password bypass and on and on.

"The Quarterback Sneak - Evading AV" amounts to Chapter 6. BDF (backdoor factory) is included in Kali and its methods of changing the functionality of normal services is described in minute detail. Next is a method for using a tool called Evade to allow another tool (Windows Credential Editor) to cloak itself from antivirus programs and snatch cleartext passwords FROM MEMORY! Neat. Then there's a discussion of Veil, a tool that hides executables from AV detection by automatically re-coding them in Python. There's SMBExec, which is a suite that can get hashes out of a domain controller, randomize or recompile things to render the observable form unfamiliar to the AV scanner (not merely changing the filename), create reverse shells, etc. He wraps up by describing some keylogging methods.

"Special Teams - Cracking, Exploits And Tricks" amounts to Chapter 7. Lots of helpful wordlist locations are given, then particular tools John (John The Ripper, JtR, JtR Jumbo) and oclHashcat are described. oclHashcat assumes use of a GPU and is mentioned as the author's favorite password cracker... although he does refer to historical use of Rainbow Tables. Specific vulnerability searching (within Kali at least) is described in the context of Searchsploit (for default queries) and the venerable BugTraq and Exploit-DB, not to mention msfconsole. Kim also gives specifics on bypassuac_injection, NetHunter (EXCLUSIVE to Kali/Offensive Security, it's an Android pen test platform) and he adds a description of building a custom reverse shell, which can get around firewalls and IDS. He also mentions three commercial products, Cobalt Strike, Core Impact and Immunity Canvas. Although Cobalt Strike costs thousands, Kim says that it's "must-have" for professional pen testers.

"Two Minute Drill - From Zero To Hero", a kind of Chapter 8, is a rundown of what's now possible to you, once you are familiar with the tools involved. From the initial stage of "discovery" (as the lawyers say), you use a fake website to spoof their Outlook Web Access page (that hole was left open for awhile on Hillary Clinton's email server, so although little was logged, it's a promising area of concern for that organization). With a Meterpreter script to add persistence on reboot for everything done so far, you'd then run PowerUser to create a new Administrator and find all the Domain Admins. Getting at least one of their passwords, the hashes are pulled from the controller and you dump the AD environment. Persist later surrepitious entry with Sticky Key and all that's left is the documentation paperwork and to invoice your satisfied customer!

I wish I fully understood everything I described above, but luckily I've got Kim's book. This provides either clear, specific steps to do what's discussed, or where that would assume previous knowledge, Kim gives links and references on where to get the knowledge. To read this book is time well spent, if pen testing is what you need to do, or need to be able to sell to a reputable actor, of course.

But This Can't Really Happen... much?

Here's the recent story on that deletion mistake, which was a marketing ploy by somebody (HOAX), but it resembled actual incidents I recall from my days working with a hosting provider:

Guy Deletes Entire Company

Yes, it's indeed possible for somebody to come up with a script that has a mass deletion command that isn't properly limited to the current directory or restricted by file privileges (in linux, Microsoft, Unix, Mac, what have you). The Admin or root permission is really powerful, and you should remember to include in the batch file or script the highly specific location(s) that any destructive command needs to work on..

But how would you architect some protections into this? One could go on and on, but the story seems to suggest a one-man operation without a staff, meaning without a professional admin, or even admin company (outsourced). Common situation. Even such an outfit as this one-man-band could benefit from two backup servers (the one in the story had an open connection and presumably its drives were mapped/mounted, therefore got the deletion command along with all other nodes that could be reached). What I'm suggesting is that, with two backup servers, you give one a regular address and the other a private, nonrouteable IP. Script a dd or other copy command from the regular server to the private server and never connect to the private server directly. That makes the first backup server a "staging area" and the second a pseudo-offline location.

Another thing to think about is to make the two backup servers physical (not virtual) servers, since if anything went wrong at those locations (and much of everything else was virtual in order to save money), you could do recovery efforts there since you'd have exclusive access to the drive(s) on those boxes, rather than on some cloud or multiple account arrangement, which would possibly overwrite data with other customer's stuff. I'm sure there are other and possibly better ideas but the simplicity of this second server as a fail-safe is compelling for this more-than-plausible scenario. And further, the backup scheme could treat the first server (staging) as holding only the last one or two backups so they'd be at hand for rebuilds and the second pseudo-offline private server would be for the archiving of a larger number of backups.

[Edit: While the above story about the guy deleting the company with a single code line has its implausibilities, I was told by a colleague about a similar situation having actually occurred a few years back. A local bank had set up much of its capabilities virtually, not just with servers fulfilling specific duties but with virtual networking as well (at least a few virtual switches). Something went wrong in accounting and the cloud provider didn't get paid on time and about 100 boxes just disappeared. That's the way it works when you miss the rent - something else happens automatically (knock at the door, late fee, whatever). In this case, I understand backups went somewhere else (hooray) but the boxes themselves had to be rebuilt and reloaded from backup by poor Mr. Admin - which was one guy, not involved with the problem but still getting the heat. Just another day on duty!]

This is the way it's done.

My congratulations to the maintainers of Linux Mint, the popular Debian-based Linux desktop distro. Two recent posts at their blog, namely

All forums users should change their passwords.

and

Beware of hacked ISOs if you downloaded Linux Mint on February 20th!

are examples of an open source maintenance organization admitting that there have been security issues AND getting word out with clear explanations and instructions for fixes with breakneck speed. Why can't many government and business organizations take heed of examples like this? Good work, guys/girls! This quick work is an example to us all.

Stuff Learned From Current And Previous Jobs

1. In a disaster, always be the first to ask "Where are the backups?". [This presumes that you are NOT assigned to do them. If you ARE assigned to backups, immediately locate the tested and complete backup needed and be ready to transfer copies to all affected locations.]

2. On ambush calls (where you answer the phone and it's four people on speaker - business to business situation - your instant hot seat at the meeting): it's not as bad as you'd think. The call proves that there's a lot of money and credentials at the table that haven't yet figured the issue out. They may bluff or bluster but underneath, they're worried. De-escalate with tone!

3. In an actual emergency, ask yourself (PRIVATELY) whose emergency is it? That's a clue as to where the solution will be.

4. Regardless of what's happening, find out how events are being documented, by whom and where.

5. Regardless of what's happening, find out how others can follow events in real time.

6. Regardless of what's happening, find out how to verify what you're being told.

7. Find out how many others (admins, users, owners, customers) have access to the work area, and if any of the above need clear and fast warning in case of application restarts, system downtime, reconfigurations, file location changes, etc.

8. Answer this: do you have permission to do the work, and how do you prove that?

9. Be diplomatic when discussing technical documentation; you might be talking to the poor soul that had to write it. Even sketchy documentation was probably very hard to obtain.

Another Clever Stall, putting off the Security Onion post yet again!

Important and weighty observations:

1. I have not (in at least 15 years) seen Microsoft Malicious Software Removal Tool find anything. No doubt this is a tribute to my top-notch adminning - or that I don't use Windows enough.

2. I see many suspicious listings for IT positions, and have for years. I wonder about, say, ones for Linux System Administrator, yet into the second paragraph creeps things like Active Directory. And it's just as unfair for that Windows Administrator offer to eventually mention that little Linux project... [translation: we need two or three people, but we'll see how far we can get the budget to go...] And it gets worse. Did you hear about the hospital that paid the ransom to get back file access this week? News stories are mentioning that healthcare companies don't spend a lot on security these days, as if we needed to be told. Looks like the two issues (security problems and responsibility creep) have the same cause: budget problems.

3. This must be a good time to be a spammer. Need a server from which to do the job? Half the servers on the air probably don't have a dedicated admin due to cost-cutting (or some outfits never having had an admin at all). There are many server/website "admins" that haven't logged into their boxes in a year. Who's watching password aging? Patching? Upgrading? Is the box even up?

4. If you need instruction, Udemy is having a sale last I heard. And they do this fairly often: many pricey courses are offered for short periods at only $10. I'm taking a few now; instructors are hit/miss but all have at least been worth the time spent.

5. And in addition to Security Onion, I'm looking at DEFT (Digital Evidence and Forensics Toolkit).

True Life Adventures In Server Support

A customer opened an issue with us, saying that his server emailed him a crash report. It did, and it was included in the incoming ticket. It was quite detailed and elegant in a way, well designed and prepared... and we had no earthly idea what it meant. The supervisor didn't know. The advanced support people didn't know. For awhile, the internet didn't – until I'd searched enough random lines in the report to come up with some hits on abrtd, and later it made sense due to the customer running CentOS. We though at first that it was a system crash from some 'abort daemon' but no, the name was an acronym: Automatic Bug Reporting Tool (daemon), a Red Hat/CentOS thing. The web hits further suggested that this tool watched for application crashes, not system crashes. We figured that uptime results would confirm that the box hadn't gone down, but the customer spent another two or three tickets finding the server password...

When the password was provided in the third ticket (I'd been the only one picking these up since I'd gotten the first one and was perversely curious), I got in and uptime confirmed that the box had been up for over 50 days. Then, I checked around the logs for something to correspond with the cryptic reports from abrtd. No luck. Fortunately, the net hits mentioned cli commands for this daemon, one of which was “list”. This was the jackpot, or at least the only promising lead. It gave six incidents (little report notices of about 5 lines each) with a timestamp and the app concerned in the fail. I recognized 5 out of six as normal for a CentOS box with cPanel, but #6 was odd: something or other ending in .py, and sitting in somebody's home directory.

Even a novice would suspect something amiss. I sprung into action and nicely suggested that perhaps he'd paid a developer to write this and might indicate to us what the script did, if anything. Also, the tool would probably flag as a problem some application activity that it hadn't been told about.

Success. The customer speculated that it was a local tool and that he'd look into it. Ticket retired.

The moral of the story is that these modern operating systems are so extensively appointed that you can be blindsided by features that no one knew about – even in a roomful of professionals. I am continually amazed at what I learn just by either answering a ticket, answering the phone or just standing around.