Leaky Third Parties & Watering Holes: Is Your Security Team Under Water?

April 15th, 2014

The April 7, 2014 NY Times article, “Hackers Lurking in Vents and Soda Machines,” surely raised the hackles of enterprise security teams worldwide. Just when you think you’ve battened down the hatches, you hear cybercriminals used a Chinese restaurant’s website as a watering hole to hack into an oil company’s network. And we already learned weeks ago that Target was breached using stolen credentials from their HVAC provider.

Not-So-Trusted Neighbors

Third party breaches aren’t new, but they’re certainly on the rise. A 2013 report estimates that 63% of data breaches are linked to a business partner’s network. As enterprise security programs get more robust, it’s easier for attackers to access a trusted neighbor’s network, which typically has more security deficiencies. Once inside, attackers can springboard into the network of the targeted enterprise.

Here are some real-world examples that we’ve seen at Damballa.

  • BYOD. At a large bank, we found 50% of their contractors’ laptops were infected and actively communicating with malware.
  • Trusted Neighbor. A large energy company was infected from a local deli’s online ordering system. We discovered that attackers embedded their exploit in a JPG image. Employees who placed orders from the corporate network were infected.
  • Non-Windows OS (yes, Mac’s do get infected). We discovered Domain Generating Algorithm (DGA) C&C activity in a University research organization that used mostly Mac devices. A few weeks after our discovery, the malware was labeled “Flashback.”

Plugging the Leaks

Shutting down your partner eco-system probably isn’t an option. So what should you do?

First, conduct a security and risk audit of your network to help uncover vulnerabilities like sketchy third-party access. Next, consider how prepared you are to detect infections that bypass your prevention controls. Finally, have an active, well-rehearsed incident response plan in place so you can take immediate action to prevent damage.

The Target breach illuminated just how difficult this process can be. Their security team was blasted by the media for supposedly failing to take action.  In their defense, Target’s prevention technology issued alerts, not proof of infection. As I wrote in a March 14 blog post, an alert by itself only tells you that unusual activity was observed. You have to corroborate the information from an alert with other data to get proof of an infection. It’s a time-consuming, resource-intensive process.

Consider this: According to Damballa’s enterprise data from Q1 2014, an average customer’s network generates an aggregate average of more than 10,000 events daily, which yields 97 active infections. Can you imagine if you got 10,000 alerts versus proof of 97 actual infections? Damballa correlates network activity through multiple detection techniques and boils down events to identify actual infections. Once the compromise is confirmed, we provide responders with indisputable evidence and risk rankings so they can prioritize their remediation.

No doubt it’s difficult for security teams to keep their heads above water when dealing with so many alerts from different attack vectors and surfaces. Do everything you can to guard your perimeter and be prepared to detect and respond to carefully-crafted attacks designed to evade the fortress walls.


Brian Foster
  CTO, Damballa

Defending Against Zero-Day Vulnerabilities, Whether There are 11 or 111

April 2nd, 2014

Last week a sandboxing technology provider published a white paper announcing they discovered 11 zero-day vulnerabilities in 2013. This week, Apple released fixes for more than 25 vulnerabilities found in Safari (that’s just one application on one operating system).

It got me thinking about the relevance of focusing on the number of zero-day vulnerabilities. Does it lead to a hamster-on-a-treadmill approach to security? Is the discovery of 11 vulnerabilities by one company in 2013 a lot or a little? Does it even matter in today’s threat environment?

Consider this: the discovery of a zero-day file doesn’t mean an endpoint was infected. Discovery of the file means that malware was observed in motion. Whether the number is 11 or 111, what matters most is whether or not the endpoint was infected.


My view is–it’s more productive to focus on what is a security problem instead of what could be a security problem. There simply isn’t enough time and brainpower for security teams to worry about events that aren’t actual threats.

Here’s an analogy. Think about a zero-day attack in terms of credit card fraud. What if your credit card company called you 11 different times to say there might be suspicious behavior on your card? What could you do with that information? Cut up the card and ask for a new one. Change all of the accounts that auto-bill to your credit card. Tell your spouse to stop using it. A lot of energy is expended without confirmation that anything nefarious occurred.

On the flip side, wouldn’t it be extremely valuable to get one phone call saying, “We know your card is being used fraudulently and we’re shutting it down.”

If no real threat exists, there is no need to know. If an actual threat is detected, the need to know is immediate.

Apply that train of thought to zero-day files. The discovery of a file itself doesn’t indicate that an endpoint is infected. At Damballa, we don’t rely on file discovery to make a determination that an actual threat has infected an endpoint. Our goal is to determine if the infection happened at all and stop it before damage is done. We detect:

  • Malware that downloads after the exploit occurs
  • Malware that downloads without an exploit
  • Network activity from malware downloaded outside the corporate environment

Last year, Damballa released a case study detailing how we stopped an unknown vulnerability we called “LazyAlienBikers” from becoming a breach. The malware targeted FORTUNE 500 companies. It had downloaded in many networks but was not tied to a known exploit, vulnerability or file.

Damballa discovered the malware <that> using successful evasion techniques to exfiltrate data, including:

  • Using SSH over HTTP ports to bypass firewall blocking of non-HTTP traffic
  • Tunneling through Web Security Gateways on port 443
  • Using a custom compile of the PuTTY client for encryption
  • Exfiltrating megabytes a day from select endpoints while other infections remained dormant

We collected overwhelming case evidence for incident responders who were able to take immediate corrective action.

As Brian Krebs noted in an article in December 2013, on any given day, the bad guys have access to an arsenal of zero-day vulnerabilities. So far in 2014, 14 known vulnerabilities have been discovered. The number doesn’t matter as much as the defense you have in place to combat advanced attacks.

I’ll close with some great advice by Stefan Frei, PhD, at NSS Labs:

  • Assume you are compromised, and that you will get compromised again.
  • Prevention is limited; invest in breach detection so that you can quickly find and act on any compromises
  • Make sure you have a process for properly responding to compromises when they do happen

Brian Foster
  CTO, Damballa

Did Target’s Security Blow it or Just Get Blown Up with Noisy Alerts?

March 14th, 2014

I’m going to play devil’s advocate and challenge the notion that Target’s security team was an epic failure.

The March 13, 2014 Businessweek article, “Missed Alarms and 40 Million Stolen Credit Card Numbers: How Target Blew It,” did a great job explaining what happened leading up to the Target breach. But it didn’t provide context about the reality of what an ‘alert’ means to a security team guarding a network as large and complex as Target’s.

Was Target negligent or did they just have too many noisy alerts to chase? What does that mean?

Here’s an analogy that may help make sense of it.

I’m sure you’ve seen shoplifting sensors at the front doors of nearly every retail store. What happens when they go off? Does the store security guard rush forward and tackle the shopper? Do the cashiers holler for help? Do iron bars descend to block the exit? The truth is nothing happens because the alarms beep all the time. Everyone in the store, including the store personnel and other shoppers, have learned to tune them out because 99.9% of the time they mean absolutely nothing.DetectionTarget-BLOG

Now step back and consider an organization the size of Target. They have more than 360,000 employees worldwide, about 2,000 stores, 37 distribution centers and a heavily trafficked retail web site. Their network is massive. A network that size may issue up to hundreds of thousands of alerts a day.

It’s essential to understand that an alert does NOT equal confidence that a device is infected. To prove infection, you need to correlate the alert with other activity or have a human being investigate the endpoint to see if it is infected.

Consider the prevention device mentioned in the Businessweek article. It monitors incoming traffic and if it sees suspicious files in motion, it executes the file in a ‘sandbox.’ Then it issues an alert.

So why didn’t the security team rush to the front door and tackle someone? Just like in the shoplifting example, the beeping alarm doesn’t mean anyone walked out the door with stolen goods.

The reality is no organization can respond to every alert. Even with a security staff of 300+ people, it’s impossible. You can’t scale any team to do that, not to mention it’s impractical for the business.

And remember an alert doesn’t equate to confidence that something is infected, much less that damage has been done.

The Businessweek article makes note that Target’s prevention device enabled it to delete malware as it was detected but Target opted to turn that feature off. While this may sound foolish, in reality if that feature were turned on it would have an astronomical effect on the business. Individual alerts have a high risk of false positives. Imagine if every time the shoplifting alarm went off, a store security guard tackled each customer that walked out the door. Do you think that would impact store operations?

So what’s the answer? As much as we would like to think there is a silver bullet that would have prevented the Target breach there simply isn’t. Today’s threat actors are highly sophisticated and always have the first move.

Enterprises should try to prevent as many threats from entering the network as possible. But they should also assume that prevention will fail.

Then what? The discussion shifts to how quickly you can detect an actual infection and respond to it.

Damballa has a different idea than most about how to approach today’s threats. We don’t just find malware and issue alerts. We rapidly identify truly compromised devices based on a case of evidence and provide certainty that the device is infected.  Security teams have confidence that when Damballa says a device is infected it is, which provides responders the ability to react promptly so they can prevent damage.

In our own labs, we find twice as many infections as the leading sandbox solution. A sandbox is a single means of detecting malware. While helpful, it doesn’t provide conclusive evidence that malware has infected a device.

Damballa uses eight different detection methods and automatically correlates real-time activity across them before verifying something is infected. Security personnel don’t have to chase alerts. Rather they receive definitive evidence about an infection. On top of that, they receive a risk score comparing the different infected devices, not a severity score. How severe is severe?

That reminds me of a line from the movie “A Few Good Men.” In a courtroom scene Tom Cruise, playing a Navy lawyer, asks Jack Nicholson’s character, playing a Marine Colonel, if the crime victim was in “grave danger.” Nicholson replied, “Is there any other kind?”

In cyber security, any malware may warrant a ‘severe’ alert but what risk does it pose to your organization specifically? There are lots of factors in play.

Damballa uses nine risk profilers to determine the actual risk based on activity of the malware, the importance of the device and threat actor attribution. When we hand off confirmed infections to a response team, we’re not only 100% confidence in the infections, but we prioritize each infected device against all other infected devices we see in the network. That information is powerful. Incident responders can now rush forward and tackle the actual bad guy before he gets out of the store with the goods.

Without first-hand knowledge of Target’s security processes and solutions, we can’t comment that Target did everything right or wrong to protect their customers; but we can appreciate the challenges that face enterprises today.

Threat actors always have the first move and they are relentless.  It’s a clear call to security vendors that we have to do better.  We must do better.  It isn’t sufficient anymore to “beep” every time something looks or seems suspicious.  It is critical that solutions start providing confidence in their detections and provide a higher level of certainty that a threat is real.

Further, security solutions must also triage the risk.  Someone walking out of the store with an unpaid stick of gum is very different than someone walking out the store with an iPad.  Security and risk teams need to not only know a device is compromised, but also what risk does it pose to the organization.  That’s the approach we take at Damballa.

Brian Foster
CTO, Damballa

2014: What the Future Holds for Breaches and Threats

January 7th, 2014

Well, as we have just closed the door on what has been a busy and interesting year for, let’s face it, both bad and good actors alike, just what exactly might be in store in 2014 when it comes to security breaches and advanced threats? Here at Damballa, we have pulled out the Tarot cards, Ouija board, crystal ball, and oh yeah, our scientific big data, to provide you with a look into the future from what we are seeing as possible signs for bigger things to come.2014-BLOG

More Sophisticated C&C Discovery Mechanisms will Emerge

If we observe the way Zeus, PushDo, ZA and other major threats evolved, it should only be expected that more sophisticated C&C discovery mechanisms — similar to P2Ps and DGAs – will be in play. The main reason behind this trend is that the WhiteHat community has to work very hard to take down a botnet that uses DGAs and P2P C&C mechanisms.   So now we are seeing more and more the use of such agile C&C communication mechanisms from the point of modern adversaries. As the criminal makes an investment of building its botnet, they certainly do not want it taken down overnight!   Thus, if the primary C&C channel goes offline, they need to have redundant C&C channels for its bots. Those channels most likely will be channels that are hard to be taken offline — in the spirit of DGAs and P2Ps.

Disruptive Technologies Will Impact the Threat Landscape

Today’s mobile technologies effectively negate the notion of perimeter-based protection for enterprises.  We have known this for some time.  In this current era, the only way an enterprise will be able to become breach resistant is through detection methodologies not tied to particular devices and operating systems.  Enterprises need to start witnessing network communications looking for illicit communications from infected devices. A holistic approach to reasoning about the entirety of one’s corporate traffic is the only way enterprises will stay relevant in the modern threat landscape.

Next Generation Security Solutions Will Rely on Big Data

There has been plenty of discussion around the futility of chasing malware. Depending on your data source, there are over a million news samples of malware a day.  Clearly any security solution that relies on seeing malware will not scale in the future.  This is why security solutions based on big data and data science will be key in the future.  Using big data and data science, future security solutions will rely on models of legitimate and illicit behavior gained from analyzing large volumes of data.

Data-driven Security Will Be Harder for the Next Generation of Threats to Evade

Information security has been a game of tit and tat between the attackers and the defenders.  As soon as the defenders deploy a new technology to stop attackers, the attackers learn how to evade those defenses.  As we have discussed in previous blogs, due to the asymmetry of information, attackers can easily learn what preventive defenses are deployed and then work to evade them.  We have seen this with IDS, IPS, NBAD ..etc..  We have seen this with present day sandboxes.  What makes data-driven security harder to evade is that in order for the attacker to know how to evade the models coming from big data security, the attacker must have access to the same set of data in order to see how the models are created.  Getting that data will not be easy for the attackers since they would need daily network data from 50% of North America internet just in order to replicate what Damballa sees.

End-to-end encryption and encapsulation techniques will be widely deployed.

Snowden made the world realize that we do not have an effective data security policy. We should soon see end-to-end encryption and encapsulation techniques being widely deployed. This will effectively kill the DPI market, however approaches to detection that are based on network objects like domain names and IPs will future-proof any network from security breaches.

Well, those are the big things we are seeing coming in the year ahead based on our research and data. We hope that by giving you some insight on what might be ahead, you and your team won’t need a crystal ball or have to guess what’s ahead but instead, be better prepared to find, respond and contain active infections before they become breaches.  May you have a happy and breach resistant New Year!

Brian Foster
CTO, Damballa


Writing MapReduce Jobs in Idiomatic Clojure with Parkour

December 9th, 2013

On the Damballa R&D team, we use Hadoop MapReduce on a daily basis to work with the Internet-scale datasets which make Damballa’s appliances tick.  We need to move quickly to adapt to the latest threats, but most of the existing tools for rapidly iterating on Hadoop are optimized for simple queries on flat data.

In order to overcome this limitation, we have developed the Parkour library for Clojure-Hadoop integration.  The Clojure programming language has been our tool of choice for quickly developing applications processing complex data.  With Parkour, we can now reap the benefits of both Clojure and Hadoop simultaneously, seamlessly writing Clojure code which runs as Hadoop MapReduce programs.

For more details, visit: http://blog.cloudera.com/blog/2013/12/write-mapreduce-jobs-in-idiomatic-clojure-with-parkour/

Marshall Bockrath-Vandegrift
Principal Software Engineer, Damballa