Everything you do in threat intelligence is about indicators or patterns. In a binary world, patterns are actually just how different indicators work together in the chain of a malicious event.

Working with threat intelligence for years now, I’ve often asked myself several fundamental cyberthreat intelligence questions:

  • What exactly is this attack and how can I identify it?
  • Is this attack description related to the technology I use?
  • Is the company I want to protect a possible target?
  • What would this attack look like in my environment?

While thinking about the right answers to these questions and creating your own attack model and hypothesis can be a lot of fun, extracting indicators from intelligence and enriching them is usually not. To speed up this task, I always prefer to write small scripts or code snippets to do these things for me, so I can focus on the fun part.

Extracting Different Types of Indicators From Cyberthreat Intelligence

All code examples below can be found on my public GitHub repository. In this blog, we will mainly look at code snippets used in indifetch.py, which is exactly what it sounds like: fetching indicators from a text or string. As a disclaimer, you should always review the regex used and not trust the code blindly. I always advise some cross-checking and keeping an eye out for a better regex or faster way to do a task — the following is just one way to do this.

Hashes

Hashes are fairly easy. They normally come in two flavors, md5 and SHA. A function that covers md5 could look like this:

def getMD5(text):
thisset = set()
md5_r = re.compile(r"([a-fA-F\d]{32})")
for item in md5_r.findall(text):
thisset.add(item)
return thisset

As you can see, we use the simple regex [a-fA-F\d]{32} to fetch the indicator out of a given text. The regex matches any character “a” to “f” and any number in a string of 32-character length. We use a Python set because items in a set are unique, eliminating duplicates right from the start.

Changing this function to cover SHA256 is an easy next step. Besides it being a totally different algorithm, the representation is the same but twice the length of the characters ([a-fA-F\d]{64}), using 64 characters instead of 32.

IP Addresses

Next up are IPs. IPv4, in particular, follows a pretty simple pattern: four numbers, none of which are higher than 255, separated by a dot.

def getIP(text):
IPlist = set()
ip = re.compile(r"\b(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\b")
for item in ip.findall(text):
IPlist.add(item)
return IPlist

We use the same style of function, as I always try to reuse code if I can. One potential problem in this situation is that, sometimes, version numbers use the same style, which can confuse the code.

URLs and Domains

The last common indicators we will cover in a function are URLs and domains. Many reports will have a large set of URLs since most malware has to connect to a command-and-control (C&C) server or exfiltrate data. In addition, with proxies and firewalls, they are often one of the easier indicators to catch.

def getURL(text):
URLlist = set()
urls = re.compile(r'(?:[a-zA-Z]|[0-9]|[[email protected]&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+')
for item in urls.findall(text):
URLlist.add(item)
return URLlist

Since you are familiar with the style by now, the regex above worked well for me, but I have also used simpler representations in the past, such as:

http[a-zA-Z0-9./:]*

Speed Up Your Threat Analysis

Python is a great tool for scripting these little code snippets and speeding up threat analysis. Some scripts can even extract indicators and make an API call to resources like the X-Force Exchange to get the current scoring of the indicator, further speeding up the process.

One regex you may want to look into is for Common Vulnerabilities and Exposures (CVE) numbers. I tend to use something simple, such as:

CVE[^\w]*\d{4}[^\w]+\d{4,}

Note: Remember to always use set() in Python instead of lists to remove duplicates right from the start. This comes in handy especially when you automate API calls as part of the script.

There are many cyberthreat intelligence tools and platforms that can do this dismantling of information for you, but it can be extremely useful to understand the magic behind the process before relying on a tool.

More from Intelligence & Analytics

RansomExx Upgrades to Rust

IBM Security X-Force Threat Researchers have discovered a new variant of the RansomExx ransomware that has been rewritten in the Rust programming language, joining a growing trend of ransomware developers switching to the language. Malware written in Rust often benefits from lower AV detection rates (compared to those written in more common languages) and this may have been the primary reason to use the language. For example, the sample analyzed in this report was not detected as malicious in the…

Moving at the Speed of Business — Challenging Our Assumptions About Cybersecurity

The traditional narrative for cybersecurity has been about limited visibility and operational constraints — not business opportunities. These conversations are grounded in various assumptions, such as limited budgets, scarce resources, skills being at a premium, the attack surface growing, and increased complexity. For years, conventional thinking has been that cybersecurity costs a lot, takes a long time, and is more of a cost center than an enabler of growth. In our upcoming paper, Prosper in the Cyber Economy, published by…

Overcoming Distrust in Information Sharing: What More is There to Do?

As cyber threats increase in frequency and intensity worldwide, it has never been more crucial for governments and private organizations to work together to identify, analyze and combat attacks. Yet while the federal government has strongly supported this model of private-public information sharing, the reality is less than impressive. Many companies feel that intel sharing is too one-sided, as businesses share as much threat intel as governments want but receive very little in return. The question is, have government entities…

Tackling Today’s Attacks and Preparing for Tomorrow’s Threats: A Leader in 2022 Gartner® Magic Quadrant™ for SIEM

Get the latest on IBM Security QRadar SIEM, recognized as a Leader in the 2022 Gartner Magic Quadrant. As I talk to security leaders across the globe, four main themes teams constantly struggle to keep up with are: The ever-evolving and increasing threat landscape Access to and retaining skilled security analysts Learning and managing increasingly complex IT environments and subsequent security tooling The ability to act on the insights from their security tools including security information and event management software…