Any woodworker will tell you that finding the right piece of wood, chopping it into perfectly sized chunks and then crafting those chunks into useful items is a complicated process. If a woodworker is making a set of chairs, for example, custom work will be needed for every piece because every tree is different. Incredible skill is needed to ensure consistency and quality craftsmanship.

Information security monitoring works the same way, more or less. Becoming a JavaScript Object Notation (JSON) formatter — and packing up your data in a new way — can reduce or eliminate the need for custom work. Being a JSON formatter can also save your security team precious time.

Logging Inefficiencies

Getting a non-standard log source into your security monitoring tool typically follows a set process. First, you find your non-standard log source and figure out how it logs and how to send those logs across your network. Next, you get those logs sending to your security monitoring tool. (Chances are they’re going via Syslog, so we’ll stick with this assumption for simplicity.) Last, you figure out how to parse out arbitrarily worded and assembled log fields using regular expressions (regex). (Depending on log complexity, this can take a while.)

There’s a massive headache here because of the inefficiency the discrepancies between different custom log sources introduce. The log’s format is essentially arbitrary and non-standardized, created at the mercurial whims of individual developers. Due to this, each time you write a custom parser using regex, it’s going to be a different and potentially substantial piece of work that will require dedicated effort from security integration experts.

JSON Fundamentals

Fortunately, there’s another way. JSON is self-describing, easy to understand and language independent.

  • JSON: This is a lightweight data interchange format that uses human-readable text to transmit data objects consisting of attribute-value pairs. Attribute-value pairs look something like this: {Date: 10/10/2018, DayofTheWeek: Monday, RequiredFood: Lasagna}. In other words? JSON is a simple, consistent syntax for exchanging and storing data.
  • JavaScript Object: This is an abstract data type composed of a collection of pairs so that each possible key appears no more than once in the collection.
  • Extensible Markup Language (XML): This is an HTML-like language that defines a set of rules for encoding documents in a format that is both human- and machine- readable. It’s a comparator to JSON in some ways.

Since the JSON format is text-only, it can easily be sent to and from a server and used as a data format by any programming language. JavaScript can convert JSON format into native JavaScript objects, which makes it easy to use JSON data like any other JavaScript object. JSON is a low-overhead alternative to XML — as JSON does not use end tags (like with HTML, these are the (end/) bits that close off elements) — which makes it shorter and quicker to read and write.

The case for JSON seems pretty compelling. However, its greatest asset (i.e., its flexible nature) also makes it situationally less suitable than XML for transferring data between separate systems or storing data that will be read by third parties. This is because it lacks some key features — in particular, schema support — which can be used to verify each piece of item content in a document. This can cause issues when moving data between systems.

The Need for Simpler Logging

The growing number of devices and application types being implemented in enterprise networks are making it increasingly difficult for log parsers to keep up with the different log formats and data values that can arise — especially from niche or proprietary custom software. Because of this, manual configuration is often needed to enable the parsing and normalizing (consistent value extraction) of logs with uncommon attributes.

When custom configuration is needed, the log’s syntax can have a significant effect on the effort required to work with them. This can include:

  • Attempting to understand the meaning of unclear values;
  • Writing countless, complex regex strings;
  • Finding a consistent anchor point — so regex remains valid with different log message types;
  • Managing the extra burden on system resources if regex is not perfectly written.

Putting JSON to Work

JSON dramatically reduces the effort needed to design and parse logs, both concerning configuration and computational resources. The key value pair (KVP) format of JSON makes it easier for humans and machines to understand what each value in the logs actually means. The JSON field names can also act as a stable anchor for writing extraction expressions. This feature reduces the likelihood of expensive or unreliable strings.

IBM QRadar is a security information and event management (SIEM) system that has a built-in collection of log parsers. In IBM QRadar, for example, it’s simple to extract log event properties created with a JSON formatter using JSON expressions.

To capture the value, one simply types: /”<field-name>”. The IBM QRadar GUI will highlight what your expression is capturing in the workspace on the right. In this case, we are capturing the sender address, so the JSON expression will be: /”SenderAddress”.

We have only scratched the surface of potential use cases of JSON and the power of becoming a JSON formatter. There are multiple ways to tackle logging outside the standard, arbitrary, one-size-fits-all format regex extraction paradigm. IBM QRadar makes it easy to take this way of working with custom log sources to the next level. Because of this, it’s worth considering this approach the next time you plan an integration.

More from Intelligence & Analytics

Email campaigns leverage updated DBatLoader to deliver RATs, stealers

11 min read - IBM X-Force has identified new capabilities in DBatLoader malware samples delivered in recent email campaigns, signaling a heightened risk of infection from commodity malware families associated with DBatLoader activity. X-Force has observed nearly two dozen email campaigns since late June leveraging the updated DBatLoader loader to deliver payloads such as Remcos, Warzone, Formbook, and AgentTesla. DBatLoader malware has been used since 2020 by cybercriminals to install commodity malware remote access Trojans (RATs) and infostealers, primarily via malicious spam (malspam). DBatLoader…

New Hive0117 phishing campaign imitates conscription summons to deliver DarkWatchman malware

8 min read - IBM X-Force uncovered a new phishing campaign likely conducted by Hive0117 delivering the fileless malware DarkWatchman, directed at individuals associated with major energy, finance, transport, and software security industries based in Russia, Kazakhstan, Latvia, and Estonia. DarkWatchman malware is capable of keylogging, collecting system information, and deploying secondary payloads. Imitating official correspondence from the Russian government in phishing emails aligns with previous Hive0117 campaigns delivering DarkWatchman malware, and shows a possible significant effort to induce a sense of urgency as…

X-Force releases detection & response framework for managed file transfer software

5 min read - How AI can help defenders scale detection guidance for enterprise software tools If we look back at mass exploitation events that shook the security industry like Log4j, Atlassian, and Microsoft Exchange when these solutions were actively being exploited by attackers, the exploits may have been associated with a different CVE, but the detection and response guidance being released by the various security vendors had many similarities (e.g., Log4shell vs. Log4j2 vs. MOVEit vs. Spring4Shell vs. Microsoft Exchange vs. ProxyShell vs.…

Unmasking hypnotized AI: The hidden risks of large language models

11 min read - The emergence of Large Language Models (LLMs) is redefining how cybersecurity teams and cybercriminals operate. As security teams leverage the capabilities of generative AI to bring more simplicity and speed into their operations, it's important we recognize that cybercriminals are seeking the same benefits. LLMs are a new type of attack surface poised to make certain types of attacks easier, more cost-effective, and even more persistent. In a bid to explore security risks posed by these innovations, we attempted to…