The Day Before Zero

I ordinarily spend a lot of my time talking about the technical aspects of threat detection and examining the tools and strategies that the bad guys are employing to subvert corporate defenses and breach their objectives, so it was refreshing last week to speak with a large bunch of C-level folks from Fortune-250 companies and to get the opportunity to step-back a little.

Talking technical is easy. Distilling technical detail, complex threats and operation nuances down to something that can be consumed by people whose responsibility for dealing with cybercrime lays three levels below them in their organizational hierarchy is somewhat more difficult. Since so many readers of this blog have strong technical backgrounds and often face the task of educating upwards within their own organizations, I figured I’d share 4 slides from my recent presentation that may be helpful in communicating how the world has changed.

The overall context of the hour long presentation was related to the paradigm change from protection back to detection – given the scope and capabilities of modern organized crime. The following slides came from the first quarter of the hour – setting the scene for how protection technologies have failed and what organizations need to do in light of that failure.

In essence, this slide talks about how that adversary has changed from old. Gone are the days of a single hacker looking to break in to an organization and toast all the systems. Sure, some of these guys still exist, but that’s not where the threat lies today by any statistical analysis. Instead, what organizations are facing is a complex ecosystem where expertise is plentiful and available for relatively low prices. Most importantly, the adversary is now a professional in every sense of the word and needs to be respected for such. Failure to do so is at your peril.

While the adversary has changed for the worse, so too has the target. Consumerization of IT and BYOD, while buzzwords in every sense of the word, really are fundamentally changing the threat landscape and the ability of organizations to combat sophisticated threats. Speaking with lots of people charged with defending their corporations from within, they really do feel powerless to combat Mac threats, Android malware, etc. or enforce application and desktop policies (for whatever that means in the world of iPads and App stores).

Everything is playing in to the bad guys hands. The devices their targets are using are varied and widespread, they roam and bridge networks, they have hundreds of applications yet few are patched in a timely manner, and the threat of personal information being leached has ensured that encryption of communications is the norm – too bad that those nosey IT security guys can inspect traffic for malicious attacks.

In essence, the onus of securing the enterprise has slipped from the corporate IT folks and landed firmly in to the hands of their enabled workforce – who happen to be poorly suited to the task.

Oh, and then there’s the “Cloud”. Not the Cloud supplying cheap processing power and high availability mission-critical applications at a fraction of the cost of legacy systems. Rather the Cloud that is the 2nd millennium USB stick – the mechanism for transporting infected files between one device and the next.

IT security departments have invested millions of dollars in their defense in depth strategies. Multiple layers of “protection” (and expense), overlapping redundancies and a continuous stream of alerts have had debilitating effects on thinly-stretched security teams.

Even if those layers of defense had been working, the “solution” for the bad guys was (and is) to “attack in depth”. The tools and techniques they now employ are multi-facetted and their complexity is hidden from the attacker. The hard work of innovation and coding was done by some expert far away, and their expertise (along with dozens of others) has been combined into a single campaign.

Last but not least, I talked about the “marginalization of protection”. My objective in this part of the discussion was to point out that trying to protect everything has never worked, and will be even less successful going forward. The consumerization of IT and the diversity of devices out there have also forced organizations (including vendors) into an area in which it is simply uneconomical to try and secure.

While effort still needs to be applied to “protecting” the enterprise, my advice is to consolidate those expensive resources around the most valuable things of the organization and only grow outwards from there if you’re successful.

In response, organizations need to assume that they are compromised and will continue to be compromised many times over, and often in many interesting ways. The onus shifts to how an organization can rapidly detect a compromise and how seamless the remediation needs to become.

I used to say that the most economical course of action was to simply reimage the computer when you were able to confirm the compromise. Nowadays that may not be quick enough, nor appropriate. Today you should reimage when your threshold of suspiciousness has been reached and, if you can’t reimage (e.g. iPads, etc.), then remotely reset the device to factory defaults and wipe any stored content so it can’t re-infect itself.

What about those critical devices – such as the CFO’s laptop – which can’t be reimaged without a lot of disruption? Let’s be clear, just because you detected one piece of malware or remote control agent on the device doesn’t mean that it’s the only one installed. And if you’re thinking you can safely remove everything related to the infection, then you’re either ill-informed or it wasn’t a threat to begin with.

Frankly, if you have critical devices that cannot be reimaged for any reason at the turn of a hat, then you’ve got bigger problems with your IT operations than mere breaches by professional criminals, and your organization needs to reevaluate its security operations at a fairly fundamental level. If a device is so critical that it cannot be recovered, it most certainly shouldn’t be a roaming laptop, accessible via the Internet, and is operated by personnel with higher than average probabilities of being targeted.

– Gunter Ollmann, VP Research

In many ways much of corporate security is a bit like dealing with those pesky odd-jobs around home. There’s always something that needs fixing, painting or screwing back in. All too often we find that many of the smaller jobs get pushed back and postponed for some reason or another despite ourselves. There’s a litany of things that should be done – like installing a doorstop behind the bathroom door to prevent the kids from slamming open the door and the handle inadvertently punching a hole through the drywall.

You know how it goes – despite the best of intentions, things get put off and then Wham! Instead of the original $5 and 15-minute effort guestimate, you now have to deal with something considerably bigger and more expensive; and there goes the entire weekend.

For the last few weeks many corporate security teams and CISO’s have been facing the same frustrated, self-induced, Homer Simpson “doh!” experience in face of the Apple Mac Flashback malware outbreaks.

They’d heard the rumblings about malware for Mac’s for years, they’ve received the glossy literature from antivirus and IPS vendors at the last few RSA conferences, and it was on their list for doing something about… soon. Next thing, they turn around and there’s ten times as many corporate Mac’s and BYOD notebooks as they thought there were, and half of them are already leaching out important files and stuff.

But why didn’t their newest anti-malware protection platforms work? Why didn’t the new tools that were meant to fill in the holes of the holes in the devices that were meant to fill in the holes of the desktop antivirus products work? Simply put, because nobody has been that interested in protecting against non-Windows 32bit malware, and the money hasn’t been there for the vendors to offer up solutions in the realm.

Take for example the latest and greatest gap-filler antivirus technology – appliance-based virtual machine malware dynamic analysis systems. It’s a mouthful, but some simply call it next generation antivirus (NGAV) or next generation IPS (NGIPS). What they’re supposed to do is automatically intercept copies of Windows 32bit executable files that are being downloaded from the Web or shuttled over email, throw them in to automated virtual machines so that the binary file is made to run, flag files that look to be malicious and, in a lot of cases, create a signature that can be deployed within the IPS component of the NGIPS solution.

So here’s the shocker, the Flashback malware infecting Apple Mac’s isn’t a Windows 32bit executable! So all those lovely shiny NGAV and NGIPS appliances being deployed out there are blissfully incapable of observing the threat (lest we forget also missing Windows 64bit malware, Android malware, iOS malware, Blackberry malware, Linux malware, etc.).

It’s not the vendor’s fault. Their products are working exactly as marketed and probably performed perfectly in the proof-of-concept and evaluation deployments as the corporate security teams chucked sample after sample of 32bit Windows malware at it. These signature-less malware detection systems just aren’t designed or built to handle the other operating system threats.

Automated dynamic analysis of malware is hard. The vast corpus of knowledge in that area is almost exclusively tied to the types of malware that affect Windows XP and Windows 2000. Handling suspicious binaries and malware that affect other operating systems and environments is more difficult, and is not quite at a level that it can be tin-wrapped and sold as an array of appliances.

The stopgap for those corporations seeking to mitigate this one specific threat (i.e. the Mac Flashback malware) who have purchased and deployed signature-less NGIPS technology is to deploy a vendor-supplied signature.

Is it just me, or is it kind of messed up that the new-fangled signature-less protection systems (which are essentially gap-fillers for signature-based network inspection engines, which are in turn gap-fillers for host-based antivirus software) require their own batch of vendor-supplied signatures to work? It’s not supposed to work this way. This is kind of like throwing a cushion behind the door after the kids have already knocked that hole in the drywall so they don’t make the hole any bigger. It neither fixes the more serious problem (i.e. the hole in the wall) nor prevents it from happening elsewhere (e.g. the other doors around the house that you haven’t extended your budding DIY home maintenance skills to).

If you were looking to deploy dynamic defense architectures and signature-less detection systems, I’d strongly advise you to examine the full spectrum of threats you’re going to face today (and next week) and choose wisely. If your organization has a mix of operating systems, devices or BYOD strategies (and don’t they all nowadays), make sure that your evaluation and testing strategy extends to these newer threats if you want to avoid another “doh!” moment and mad scrabbling for post-breach fixes.

– Gunter Ollmann, VP Research

Most of the good thrillers I tend to watch have spies and assassins in them for some diabolical reason. In those movies you’ll often find their target, the Archduke of Villainess, holed up in some remote local and the spy has to fake an identity in order to penetrate the layers of defense. Almost without exception the spy enters the country using a fake passport; relying upon a passport from any country other than their own.

Like any good story, there’s enough truth to the fiction to make it believable. Take the real-life example of the hit squad that carried out the assassination of a Hamas official in Dubai early 2010. That squad (supposedly Israeli) used forged passports from the United Kingdom, Ireland, France and Germany.

So, with that bit of non-fiction in mind, why do so many people automatically assume that cyber-attacks sourced from IP addresses within China are targeted, state-sponsored, attacks? Are people missing the plot? Has the Chinese APT leapfrogged fact and splatted in to the realm of mythology already?

If you’re manning a firewall or inspecting IPS log files, you can’t have missed noticing that there’s a whole bunch of attacks being launched against your organization from devices hosted in China on a continuous basis. A sizable fraction of those attacks would be deemed to be “advanced”; meaning that as long as they’re more advanced than the detection technology you’re using, they’re as advanced as they need to be to get the job done.

Are these the APT’s of lore? Are these the same things that government defense departments and contractors quake in their boots? There’s a simple way to tell. If what you’re observing in your own logs shows the source as being from a Chinese IP address it almost certainly isn’t.

Yes, there’s a tremendous amount of attack traffic coming from China, but this should really be categorized as the background hum of the modern Internet. China, as the most populous country on the planet, isn’t exempt from having more than its fair share of Internet scoundrels, wastrels, hackers and cyber-criminals — spanning the full spectrum of technical capability and motivations. Even then, the traffic originating from China may not be wholly from criminals based there — instead it may also contain attack traffic tunneled through open proxies and bot infected hosts within China by other international cyber-criminals.

When we’re talking about cyber-warfare and state-sponsored espionage, we’re not talking about a bunch of under-graduate hackers.

Just about every country I can think of with a full-time professional military force has been investing in their cyber capabilities – both defense and attack. While they’re not employing the crème de la crème of professional hacking talent, they are professional and have tremendous resources behind them, and they follow a pretty strict and well thought-out doctrine. If you’re in the Chinese Army and have been tasked with facilitating a particular espionage campaign or to aid a spy mission, the last thing on earth you’re going to do is to launch or control your assets from an IP address that can be easily traced back to China. Anywhere else in the world is good, and an IP address in a country that your foe is already suspicious of (or fully trusting of) is way better.

Don’t get me wrong though, I’m not singling out the Chinese for any particular reason other than most readers would be familiar with the hoopla of Chinese APTs in the media. Any marginally competent adversary is going to similarly launch their attacks from a foreign source if they’re planning on maintaining deniability should the attack ever be noticed – just like those spy tactic of using foreign passports.

So, if you’re so inclined, how are you going to get access to foreign resources that can proxy and mask your attacks? Elementary my dear Watson, there’s a market for that. First of all there’s a whole bunch of free and commercial anonymizing proxies , routers and VPN’s out there – but they may not be stable enough for conducting a prolonged campaign (and besides, they’re probably already penetrated by a number of government entities already). Alternatively you could buy access to already compromised systems and hijack them for your own use.

Over the last five years there have been a bunch of boutique threat monitoring and threat feed companies spring up catering almost exclusively to the needs of various national defense departments. While they may offer 0-day vulnerabilities, reliable weaponized exploits and stealthy remote access Trojans, their most valuable offering in the world of state-sponsored espionage is arguably the feed of intelligence harvested from the sinkholes they control. Depending upon the type of sinkhole they’re fortunate to be operating, and which botnet or malware campaign that happened to utilize the hijacked domain, they’re going to have access to a real-time feed of known victim devices from around the world, copies of all the data leached from the victims by the malware and, in some cases, the ability to remotely control the victim device. Everything a cyber-warfare unit is going to need to hijack and usurp control of a foreign host, and launch their stealthy attack from.

Now, if I was say working within the cyber-warfare team of the French Foreign Legion or perhaps the DGSE (General Directorate for External Security) and interested in gathering secret intelligence about the investment Chinese companies are making in sub-Sahara mineral resources, I’d probably launch my attack from a collection of bot-infected hosts located within US or Australian universities. The security analysts and incident response folks working at those Chinese companies are probably already seeing attack traffic from these sources off-and-on, so my more specialized and targeted attack would unlikely raise suspicion. Should the targeted attack eventually be discovered, the Chinese would simply blame the US and Australian governments – rather than the French.

Having said all that, you’ve probably seen movies with double-agents in them too. And it’s entirely possible that someone hair-brained enough would argue that China launches attacks from their own IP space because everyone knows that you shouldn’t, and therefore the assumption needs to be that attacks launched from China are clearly not from the Chinese government – while they are in fact. How very cunning. Now there’s a twist for the next spy movie.

– Gunter Ollmann, VP Research

Like the scene of a movie in which a biblical character holds back the mighty sea and is about to release the tide against his foes, BYOD has become a force of nature poised to flood those charged with keeping corporate systems secure.

Despite years of practice hardening systems and enforcing policies that restrict what can and can’t be done within the corporate network, businesses are under increasing (if not insurmountable) pressure to allow a diversifying number of personal devices to connect to their networks and be used for business operations. Bring your own device (BYOD) is the most intrusive trend that security teams have had to face for quite some time.

BYOD

Unlike other business changes over the years that caused security teams to reevaluate their policies (such as allowing remote users to VPN in to the corporate network or enabling webmail facilities for roaming users), BYOD is being driven by all levels of the corporate hierarchy simultaneously. And it’s forcing new changes in the way organizations conduct business and seek to secure themselves.

BYOD is directly forcing the hand of security teams; and those that don’t (or can’t) accommodate the change are in for a very rough ride indeed.

Organizations that have embraced the approach – allowing employees to bring in their personal devices and engage with business systems – appear to have reaped rewards ranging from increased productivity, through to a lowering of capital expenditure within their IT departments. BYOD is affecting all walks of life. For example:

• Out-of-hours system monitoring and alerting through Android applications that can be trivially loaded on to an employee’s Smartphone.
• Larger pockets being added to medical staff’s lab coats and smocks to accommodate the iPads they’re increasingly carrying around.
• Shared use of cloud storage facilities as employees jump back and forth between personal and corporate devices throughout the day.

Not all businesses have embraced a BYOD culture the same way. In the majority of organizations I deal with, the general security strategy is to treat the device as “untrusted” – typically only allowing the user of the device to connect to the Guest or dirty wireless networks and limiting access to those services or business applications that can ordinarily be accessed remotely (e.g. through a VPN). Meanwhile, a handful have gone ‘whole hog’ as it were, and are doing away with corporate supplied computing devices; instead they’re offering to subsidize the employee’s purchase and provide a list of “minimum” security standards for the device.

We are in a transitional period with respect to BYOD strategies and there is a lot of experimentation as organizations strive to achieve a new balance between security and convenience. As such, the security posture of an organization needs to take into account the continuous change going on about it. While it’s been a common declaration within the security community that you can’t protect the end-point from a determined attacker, as device ownership slips from the hands of the corporate entity into the hands of the employee, so too does the onus for protecting it.

For many organizations the frontline in security for the last two decades has been protecting computers with host-based defenses. Sure, there’s been investment in perimeter defenses, but the war between the cybercriminals and their prospective victims has been happening with the operating systems, web browsers and applications of the end device. As such, with control of the end-point device slipping out from control and oversight of corporate security teams, an added emphasis is being placed upon two critical security approaches – securing the core (centralized) intellectual property and data of the organization, and rapidly identifying devices that have already been compromised.

Organizations with a mature security strategy flexible enough to accommodate BYOD demands have pursued an approach in which it is assumed that the user’s device is likely (if not already) compromised and under control of an external criminal entity. As such, they have myopically focused their attention on securing the servers that really matter to the business and are securing the system and repositories that govern or track the data itself. In parallel, they’ve deployed systems that alert and identify devices that are acting suspiciously or are positively identified as being usurped by professional crimeware, and take immediate, automatic steps to restrict and cauterize the threat.

BYOD has forced a paradigm change in the way businesses approach and enforce security within their organizations. Security teams within organizations that continue to resist the adoption and use of personal devices (whether they be personal laptops, Smartphones, tablets or X-Box) are fooling themselves if they think they can hold back the tide. Security consolidation and threat alerting are the ropes they need to grasp.

– Gunter Ollmann, VP Research

This morning, Global Payments held a conference call with investors and analysts covering their earlier breach announcement and projected earnings. Global Payments had also released an update advisory yesterday stating that “the company believes that the affected portion of its processing system is confined to North America and less than 1,500,000 card numbers have been exported” and that only Track 2 card data may have been stolen.

In discussing the breach, Paul Garcia, Chairman and CEO of Global Payments, reiterated that the investigation is ongoing, but that the 1.5m stolen card details likely represents an upper bound to the loss and that it only affected a “handful” of North American servers (i.e. this was not a Merchant breach). At this point, they are not aware of any fraudulent transactions related to the data theft.

Obviously, given the fact that they self-reported a breach, Global Payments is no longer Visa PCI certified and must now attempt to re-earn their ROC (Report on Compliance). Although they’re not Visa PCI certified, that doesn’t mean that they cannot process Visa cards – rather that, by being non-compliant, they will be liable for fines and additional losses. When asked during the call as to the likely charges and liability of the breach, listeners were reminded several times that the investigation is continuing and that the company has sufficient insurance to cover prospective liabilities. It was stated that Mastercard may take similar PCI certification actions.

I thought it was interesting that Global Payments had received assurances from competitors that they wouldn’t capitalize on the breach – since any one of them could be similarly affected in the future (if not already breached, but undetected so far). I’m not sure how credible that is, and I’d be surprised that some of the competitor’s sales folks aren’t already independently using the breach to further their own agendas.

Global Payments stressed that, contrary to rumors, this is the first breach that the company has suffered. The breach itself is believed to be contained and was picked up by their server data monitoring and breach detection tools – “just not well enough” (no hints were made as to the nature of the technology deployed).

So, while the forensics investigations continue, what does it all mean? Based on the information disclosed thus far, it sounds like Global Payments is doing everything the right way. They disclosed as soon as they had enough information and confidence in their discoveries to do so. They’ve been using data monitoring tools to spot breaches – albeit these controls proved to be insufficient to stop the threat and don’t sound like they were real-time reporting enabled. They’ve pulled in experts to help them get to the bottom of the breach. And they’re aware of the business consequences – having taken out sufficient insurance to protect against associated liabilities. What’s left?

Last week a number of 10,000,000 had been thrown out as to the size of the theft. It now appears that 1,500,000 cards were stolen. No discussion was provided as to what other data had been exposed (i.e. no “evidence” that it had actually been stolen). Regardless, while 1.5m is less than 10m, it’s still a damned big number and it will cost the card distribution agents quite a bit of money to clean up and reissue cards – all of which Global Payments will need to cover. I think that lessons have been learned from the big data breaches like TJX, but it would appear that the cost of a breach is largely independent of the number of cards actually lost.

Global Payments has been deliberately cautious in revealing any details as to how the incident occurred and the nature of the systems that failed to protect against the penetration or alert to the breach. I’d expect that time will shed more light on the attack vectors. It is important that such details are exposed as and when it is prudent to do so. While Global Payments is a multi-billion-dollar enterprise, there are still hundreds of other card clearing houses around the world that could benefit from detailed disclosures of the incident so that they could construct better defenses. While these may be competitors to Global Payments, we – as in you and I – are the potential victims of their inadequate defenses and I’d like assurances that they’re doing better than they are today.

– Gunter Ollmann, VP Research