close
close

After trial, Splunk and Cribl find themselves in data pipelines

Posted on by angelo
After trial, Splunk and Cribl find themselves in data pipelines

As the growth of observability data continues, driven by the hype of generative AI, log management rivals Splunk and Cribl find themselves on a new battlefield – and this time, it’s is more on Cribl’s ground than on that of Splunk.

The relationship between the former partners deteriorated in 2022, when Splunk filed a lawsuit against Cribl, alleging copyright and patent infringement. Cribl’s business, founded in 2017 by a Splunk alum, was initially based on log management software that reduced the amount of data sent to Splunk, where users had to pay for data ingestion, thereby reducing costs. costs to Splunk users and revenues. . In April, that lawsuit, which had evolved into a case involving violations of licensing and partnership agreements, went to trial in court, where a jury found that Cribl violated the terms of the Splunk Terms for Splunk Enterprise and awarded Splunk $1 in damages.

Much has changed in both companies’ business models since the lawsuit was filed, even before Cisco announced plans to acquire Splunk in September 2023. Cribl has expanded beyond log management to s ‘expand to data pipelines, federated search and its own data lake. Splunk also adjusted its pricing starting in 2021, with options to pay per workload rather than per data ingestion, as well as its own federated search that encompassed data stored in cheaper Amazon S3 buckets.

As of this month, both companies have been touting new product features aimed at making data management more efficient and inclusive of disparate data sources. Splunk expanded its Unified Data Platform at its annual .conf user conference to include Pipeline Builders and an ingestion processor for Splunk Cloud, as well as federated analytics capabilities.

At its own conference that same week, Cribl released its first features since the lawsuit verdict, which included performance improvements and expanded third-party support for its federated search. Both companies also launched new AI assistants.

The similarities have not been lost on industry observers.

“Splunk’s Cribl-like capabilities are exciting to see,” said Gregg Siegfried, an analyst at Gartner. “Maybe too little, too late, but I welcome them to the telemetry pipeline business. It’s becoming a popular industry.”

Splunk data management and Cribl search

There are many similarities between the Splunk and Cribl data pipelines and federated search, and both rolled out new features last week intended to improve the performance of federated data analytics.

“Last year, we launched federated search so you can remotely search Splunk S3 data and correlate it with your Splunk data… for historical and audit use cases,” said Faya Peng, Vice President of Product Management at Splunk, speaking at a .conf conference. opening speech. “Now, with federated analytics, starting with Amazon Security Lake, you can selectively retrieve data…and integrate it into a short-term index. This enables more powerful use cases like monitoring and ad hoc investigations. »

Cribl has added Real Time Fast Query, a feature that will extend Cribl Search support beyond archive data in object stores such as S3 to access new data with faster response times. The product now also supports querying across multiple indexes, something a potential customer was waiting for before replacing Amazon OpenSearch with Cribl Search.

“Prometheus data is (also) something we want to test,” said Bob Chen, director of infrastructure engineering at iHerb, an online retailer of health and wellness products in Irvine, California. “(It) looks like we’re ready to test it again and I just need to find the bandwidth to do it.”

However, Splunk and Cribl take unique approaches to accessing different data sources within search and analytics workflows. Cribl’s Search documentation describes a process that does not require the selective retrieval and indexing described by Splunk’s Peng.

“Cribl Search… (can) seamlessly analyze all data directly at its source,” the website states. “Cribl Search allows users to search and analyze data wherever it resides – from debug logs at the edge to data archived in cold storage – using a single… query language.”

A Cribl Search datasheet linked on the website adds that “(it) searches across multiple data stores and multiple data types… (using) a ‘search then transmit’ model instead of the traditional ‘transmit then to research “.

As of this month, Cribl Search supports more data sources than Splunk Federated Search or Federated Analytics, including time series databases and cloud data warehouses such as Prometheus, Snowflake, Elasticsearch , Azure Data Explorer and AWS OpenSearch.

Cribl and Splunk aren’t the only ones trying to take advantage of the data pipeline concept. Other observability providers, including Dynatrace and Mezmo, have introduced similar offerings over the past 18 months. Splunk is still in its early stages and is positioning Federated Analytics primarily for its Splunk Enterprise Security platform rather than observability so far, according to Andi Mann, global CTO and founder of Sageable, a consulting and consulting firm in technology in Boulder, Colorado, who served as a senior technology advocate at Splunk from 2015 to 2021.

Cribl is already well ahead of Splunk in its niche and has been for some time.

Andi MannGlobal CTO and Founder, Sageable

“It looks like Splunk is finally trying to recreate a Cribl-like feature,” Mann said. “Splunk should have offered Cribl as a feature five years ago, but missed the opportunity during product planning and was unable to resolve the issue in court. Today, Cribl is already well ahead of Splunk in its niche, and has been for some time.

A customer of both vendors said their federated search and analytics capabilities could potentially be complementary, given their different methods of accessing data. He said Splunk is likely to offset revenue generated from data ingestion by charging for ingestion processing and analysis based on its Splunk Virtual Compute (SVC) workload pricing.

“Splunk enabling federated search on non-Splunk pooled data is a really good idea, but goes against their ingestion-based licensing model,” said Steve Koelpin, Splunk principal engineer for a Fortune 1000 company. from the Midwest. “Now if customers are using SVC, that’s a different story. Federated search can be very slow and will eat up SVC (resources).”

That resource cost could be worth it, depending on where the data resides, Koelpin added.

“It’s best to clean, transform, structure and index the data in Splunk, and if that faithful copy is needed, just store a copy in S3,” he said. “But if you (have) a lot of high-fidelity copies of data and you want to do a retroactive security investigation but you don’t want to index 100 TB of data, then (Cribl) Search is the way to go.”

In the age of AI, Cisco/Splunk say size matters

Cribl has a lead over Splunk in data pipelines, but its AI assistant, Cribl Copilot, remains limited in its first release, analysts say.

Jon Brown, Analyst, Enterprise Strategy GroupJon Brown

“Cribl has done a great job of sticking to its knitting, staying focused on the observability data pipeline problem,” said Jon Brown, an analyst at TechTarget’s Enterprise Strategy Group. “(Cribl) Copilot is a ‘me too’, and quite lightweight. (Cribl maintainers) have openly admitted that they can’t really get it to work properly, so they have limited functionality for now. J I appreciate their transparency.”

A Cribl spokesperson explained further in an email to TechTarget Editorial:

“As a first release, the best part of Copilot is how it continually improves and learns as you use it. Today, if you ask him something he doesn’t know, he’ll just admit that he doesn’t know instead of providing misleading information “We actually think it’s very important. We know we’re dealing with real-world environments in mission-critical applications, so rather than allowing AI to hallucinate, we’d rather it stop. “

Cribl Copilot is limited to Cribl.Cloud, although support for on-premises versions is on the near-term roadmap, the spokesperson said. The company’s website describes it as primarily focused on easier configuration of Cribl’s products, pipelines, and search queries, rather than analyzing the observability data itself for insights into application and infrastructure performance.

Splunk has rolled out a setup wizard for its IT Service Intelligence product that performs a similar set of functions, along with drift detection and alert threshold analysis, and an AI-driven natural language interface for its search processing language Splunk. These tools are generally available and have revealed another new AI assistant in Observability Cloud in private preview, as well as an AI assistant in security that will reach private preview in August.

Cribl can route data to other AIOps and security automation tools, but Splunk offers its own, as well as new integrations with Cisco security and networking products. Unlike the current version of Cribl Copilot, Splunk’s AI Assistant in Observability Cloud automates the analysis of observability data and system documentation for troubleshooting purposes, and provides step-by-step recommendations for resolving issues, according to the main .conf demos.

As an emerging company, Cribl is naturally at a different scale and stage of maturity than Splunk and Cisco and, as such, is an increasingly ripe acquisition target, according to Brown and Mann .

“Given the direction Cisco is taking with Splunk, Cribl appears to have a sustainable competitive advantage and continues to innovate at a premium, although ultimately it will likely end up as a feature of the company’s offering. “another mega-vendor,” Mann said. said.

Time will tell whether that mega-vendor could actually be Cisco, but Brown said a venture capital firm or holding company is also a possibility.

“I worry about the future of this company because it seems like an obvious buy for holding companies or venture capital firms for software companies, which could destroy some of the uniqueness of this group of committed people,” he said. “Uniqueness like free training and per-use licensing instead of more commonly used metrics like stored data or transformed data.”

Beth Pariseau, senior editor for TechTarget Editorial, is an award-winning veteran of IT journalism covering DevOps. Do you have any advice? Send him an email or contact @PariseauTT.