Log management for security-related purposes has become one of the biggest use cases for big data solutions, a survey by the SANS Institute shows.

Big Data Solutions

SANS earlier this year polled some 206 IT managers from medium and large organizations on their experience implementing and using big data technologies for addressing business issues. More than 40 percent of the respondents were from organizations with more than 10,000 employees, and about 52 percent had jobs that related directly to security. The results were compiled in the survey “Enabling Big Data by Removing Security and Compliance Barriers.”

The responses showed that log management applications are a big driver of big data solutions. More than 50 percent of the organizations that have deployed big data technologies are using them to manage log data gathered from the myriad devices and applications running on their networks. Storage and archiving were two other leading use cases for big data platforms, the survey indicated.

“Respondents with big data implementations are currently focused on use cases that are infrastructure-oriented,” SANS said in the report, which was sponsored by Cloudera. Examples include log management, data archival and operational data stores, the report noted.

Machine-generated logs can include data from Web applications, the servers that support these applications, mobile and network devices and other sources.

Invaluable Data

Security analysts consider such data invaluable in helping organizations monitor, troubleshoot and debug issues in network components, systems and applications. Many organizations use log data in forensic analysis applications to identify the root causes for problems on their networks. Banks and financial services companies rely heavily on log data for antifraud purposes and for ensuring that activities on their networks meet regulatory compliance requirements.

Large organizations can generate huge volumes of raw log data on a daily basis. In order to analyze the data and derive value from it, they must be able to ingest, organize, store and retrieve the information in as efficient a manner as possible. Big data solutions can help address some of these issues by offering organizations the scalability and analytics tools they need.

Spiking Interest

The responses showed that interest in big data solutions is spiking across industry sectors, with 27 percent claiming they already have a big data application in production, 28 percent saying they have one in development and another 28 percent planning to deploy a big data application in the next two years. Nearly three-quarters of organizations store personally identifiable information (PII) and sensitive business content in big data environments. As a result, they’re putting greater focus on access controls, encryption and similar security controls for protecting the data, the survey showed.

While log management for security purposes appears to be the most popular use case for the technology, different industries have different use cases for big data solutions, the SANS survey showed. For government organizations, some typical uses cases included fraud detection, compliance and regulatory analysis and climate analysis and weather prediction. Meanwhile, compliance and regulatory analysis was a major use case for financial services firms, followed by risk analysis and management, fraud detection and security analytics and CRM and customer loyalty programs. Within the IT industry, the biggest use case was security analytics and log management.

Big data solutions are just one of the methods organizations can employ to balance the delicate scales of security and compliance. Using analytics to simplify log management — among its many other applications — streamlines security protocols, organizes information and makes it easier for businesses to manage their risk and compliance.