Elasticsearch / ELK Stack

What it is

Elasticsearch is a distributed, JSON-document-oriented search and analytics engine built on Apache Lucene, originally released in 2010. In the observability context it is almost always discussed as part of the “ELK stack” — Elasticsearch for storage and full-text search, Logstash (or, more commonly today, Beats/Elastic Agent) for ingesting and parsing logs, and Kibana for visualization and dashboards. Unlike label-indexed systems, Elasticsearch indexes the full content of every document, which enables rich, ad hoc full-text and structured queries across arbitrary fields at the cost of significantly higher storage and compute overhead. The company behind it, Elastic (formerly Elasticsearch BV), now describes itself as “the Search AI Company,” reflecting its expansion into vector search, RAG, and AI-assisted search alongside its long-standing logging, observability, and security (SIEM) product lines.

Elastic’s licensing has changed more than once. In 2021, Elastic moved Elasticsearch and Kibana off the OSI-approved Apache 2.0 license to the Server Side Public License (SSPL) and its own Elastic License, largely in response to AWS offering a competing managed Elasticsearch service without contributing back. In September 2024, Elastic reversed course and added the OSI-approved AGPLv3 as a third licensing option alongside SSPL and the Elastic License — meaning Elasticsearch and Kibana are, as of the current release, licensed under AGPLv3, SSPL, or the Elastic License (user’s choice), restoring a legitimately open-source option after three years without one.

Why teams choose it

• Full-text search quality is hard to beat. Elasticsearch’s Lucene-based indexing and query DSL support relevance scoring, fuzzy matching, and complex aggregations that label-only systems like Loki simply cannot do.
• One engine, many use cases. The same cluster commonly backs log analytics, application/product search, security (SIEM) use cases, and increasingly vector/semantic search, which lets some organizations consolidate on fewer systems.
• Kibana is a mature, capable UI, particularly for security and log investigation workflows, with a large ecosystem of prebuilt dashboards and detection rules.
• Cost and operational weight are real trade-offs. Full-content indexing means Elasticsearch clusters typically need far more storage, memory, and tuning attention (shard sizing, mapping design, hot/warm/cold tiering) than a label-indexed logging system at equivalent log volume — this is the main reason teams look elsewhere purely for high-volume log storage.

Pricing model

Elasticsearch and Kibana are open-source and self-hostable, licensed under AGPLv3, SSPL, or the Elastic License. Elastic also sells a managed SaaS offering (Elastic Cloud) and self-managed subscription tiers that add security, machine learning, and enterprise support features on top of the open core, generally priced on a combination of resource usage and feature tier, with a free tier available for small deployments and trials.

Alternatives

For log-specific workloads where full-text indexing isn’t required, Grafana Loki offers a much cheaper label-indexed alternative. Splunk is the long-established commercial competitor for log management and SIEM, generally at higher cost but with a mature query language (SPL) and enterprise tooling. OpenSearch, the AWS-led fork created in response to Elastic’s 2021 license change, remains a widely used drop-in-compatible alternative for teams that want an Apache 2.0-licensed Elasticsearch equivalent.