In my continuing series of keynote recaps, I’d like to discuss two talks that center on using big data analytics to offset the uncertainty of advanced cyber attacks. The first is the keynote from the fourth annual Hack in the Box (HITB) security conference in Amsterdam by Eddie Schwartz (@eddieschwartz), who also coauthored the related paper Big Data Fuels Intelligence-Driven Security. The second is an RSA 2014 talk, Securing the Big Data Ecosystem by Davi Ottenheimer (@daviottenheimer).

The Case for Big Data

Schwartz begins his keynote by referencing the movie and book Moneyball, using it to illustrate that fighting against advanced threats means we need to be looking at nontraditional metrics and data sets. He raises an interesting point that, traditionally, we have been spending a lot of time and financial resources on perimeter defense, but there is a lot of opportunity to collect data and react once the attacker has breached initial defenses and is busy in lateral movement, escalation or exfiltration. We need to be thinking about what sort of queries we can make to identify these activities.

One of the challenges we have in identifying threats is that it is not about spotting an obvious looking problem; advanced threats are hard to identify and will blend well into the background. Big data analytics can help us with that. Using big data, we can start to see what is deviating from the norm. These problems can manifest themselves in a number of ways, including across domains, IP origins, checkout times, website browsing patterns, pipe names (for something at an OS level) and more.

As we move into advanced threat analysis using big data, we are not only trying to recognize good versus bad, but we are examining the critical versus noncritical; what needs to be acted upon now versus what can wait. If we associate a dollar value to risk, what will we lose if we don’t act immediately?

Challenges in Using Big Data

Both talks mention several challenges with using big data to predict or identify security threats. They include the following:

  • Acquisition of skilled staff to deploy, analyze and grow the big data environment, or partnering with those who have it
  • Comprehensive data collection and visibility
  • Normalization of data coming form different sources, platforms, geos, etc.
  • Premining
  • Use of new parsers to analyze the new types of data being collected

Another challenge that Ottenheimer identifies is the use of controls in big data environments and securing big data itself. This needs a whole new approach as we work on continuous collection of massive data sets, sharing them within and outside organizations, and across international boundaries. Not only do we have to care about availability of data, but also about securing the data itself.

What About the Traditional Checklist Model

Schwartz mentions that although the traditional checklist model — white/black, good/bad — gives us certainty, we can’t ignore the uncertainty. We can use the collection, processing and querying of big data to deal with that uncertainty. At the same time, this does not mean that we go to the other extreme and throw out the traditional model completely. Ottenheimer gives the analogy of fighter pilots using checklists during takeoffs. They have a lot of value, but when an emergency situation arises, or they are engaged in fight, then more intelligent methods are needed. Intelligence driven security is the model that allows you to look at a huge collection of data in innovative and intelligent ways to identify misbehaving assets as soon as possible.


One of the points that Schwartz mentioned is to start moving from big data knowledge to big data deployment. He also talks about outsourcing and partnering for some of these roles, and says that companies should start asking partners and vendors about how they are adopting nontraditional analytics.

More from CISO

Who Carries the Weight of a Cyberattack?

Almost immediately after a company discovers a data breach, the finger-pointing begins. Who is to blame? Most often, it is the chief information security officer (CISO) or chief security officer (CSO) because protecting the network infrastructure is their job. Heck, it is even in their job title: they are the security officer. Security is their responsibility. But is that fair – or even right? After all, the most common sources of data breaches and other cyber incidents are situations caused…

Transitioning to Quantum-Safe Encryption

With their vast increase in computing power, quantum computers promise to revolutionize many fields. Artificial intelligence, medicine and space exploration all benefit from this technological leap — but that power is also a double-edged sword. The risk is that threat actors could abuse quantum computers to break the key cryptographic algorithms we depend upon for the safety of our digital world. This poses a threat to a wide range of critical areas. Fortunately, alternate cryptographic algorithms that are safe against…

How Do You Plan to Celebrate National Computer Security Day?

In October 2022, the world marked the 19th Cybersecurity Awareness Month. October might be over, but employers can still talk about awareness of digital threats. We all have another chance before then: National Computer Security Day. The History of National Computer Security Day The origins of National Computer Security Day trace back to 1988 and the Washington, D.C. chapter of the Association for Computing Machinery’s Special Interest Group on Security, Audit and Control. As noted by National Today, those in…

Emotional Blowback: Dealing With Post-Incident Stress

Cyberattacks are on the rise as adversaries find new ways of creating chaos and increasing profits. Attacks evolve constantly and often involve real-world consequences. The growing criminal Software-as-a-Service enterprise puts ready-made tools in the hands of threat actors who can use them against the software supply chain and other critical systems. And then there's the threat of nation-state attacks, with major incidents reported every month and no sign of them slowing. Amidst these growing concerns, cybersecurity professionals continue to report…