Elasticdump spends its working life moving documents out of one Elasticsearch index and into another index, a file, or an S3 bucket, and that work raises a question that comes up surprisingly often: if data is being extracted, adjusted, and loaded somewhere, isn’t somebody in this picture doing ETL — and isn’t that somebody Elasticsearch itself? The confusion is understandable. Elasticsearch sits in the middle of a stack whose name sounds like a pipeline, it ships a feature with the word “pipeline” in it, and tools in its orbit can genuinely change documents while copying them.
The short answer is no. Elasticsearch is a search and analytics engine: its job is storing, indexing, and searching large volumes of data, and that is the whole of its job description. The ETL work in the Elastic ecosystem — pulling data out of other systems, reshaping it, and delivering it where it needs to go — belongs mainly to Logstash, with a smaller, in-cluster version of it living in Elasticsearch’s ingest pipelines. This article walks through what ETL actually covers, which part of the stack does which job, and where a data-movement utility like elasticdump honestly fits, because having a transform option is not the same thing as being an ETL platform.
What ETL actually means
ETL stands for extract, transform, load, and each of the three words describes a distinct job:
- Extract means reading data out of a source system: a relational database, an application log file, a message queue, an API, a stream of events. The sources are arbitrary and usually have nothing in common with the destination.
- Transform means changing the data on the way: parsing unstructured log lines into fields, renaming or removing fields, converting dates into a consistent format, looking up a geographic location from an IP address, joining or enriching records with data from somewhere else.
- Load means writing the result into a destination — a data warehouse, a search index, another database — in batches or on a schedule, with the process managed and repeatable.
A tool earns the ETL label when it covers that whole arc: connectors to sources it does not control, a real transformation layer, and managed, often scheduled, loading into destinations. The connecting, processing, and load management are the product. An ETL server’s daily work is exactly that operational burden — connecting to sources on a schedule, processing transformations as data flows, and managing loads in batches, including the unglamorous parts like retrying a failed batch without duplicating the last one. Storage engines, by contrast, focus on speedy storage, indexing, and search once data has arrived. Judged against that definition, the question answers itself fairly quickly — but it is worth seeing exactly where the boundary falls, because Elasticsearch does more data-shaping than most storage engines.
Why people assume Elasticsearch does ETL
The main culprit is the stack’s own name. ELK — Elasticsearch, Logstash, Kibana — reads like three stages of one pipeline, and it is natural to assume the letters map onto extract, transform, and load in some order. They do not. In the classic ELK stack, Logstash is the pipeline: it collects, parses, and ships data. Elasticsearch is the destination the data lands in, the store-and-search end of the arrangement. Kibana is the visualisation layer on top. Elasticsearch’s presence in a pipeline does not make it the pipeline, any more than a warehouse becomes a delivery network because trucks keep arriving at it.
The second source of confusion is real functionality. Elasticsearch can transform documents at the moment they are indexed, through ingest pipelines, and it can query and aggregate data in ways that feel like processing. Both are genuine capabilities, and both stop well short of ETL — for reasons the next two sections make concrete.
The tool that actually does ETL here: Logstash
Logstash is the pipeline tool in the Elastic ecosystem, and its architecture is the classic ETL shape. It is a server-side pipeline with three stages of its own:
- Inputs pull data in from a long list of sources — files, JDBC databases, Kafka, syslog, HTTP, TCP, Beats shippers, and many more. Its plugin catalogue runs to over 200 plugins, most of them existing purely to connect to things.
- Filters transform data in flight: grok patterns parse unstructured log lines into named fields, geoip filters enrich an IP address with location data, mutate filters rename, remove, or convert fields, and there are dedicated filters for parsing JSON and normalising dates.
- Outputs deliver the result to a destination — most commonly Elasticsearch, but not only Elasticsearch. One pipeline can route data to multiple destinations, which is a defining ETL behaviour.
That combination — arbitrary sources, a transformation layer with enrichment, and fan-out to destinations — is what a real ETL tool looks like. Notice what it has that Elasticsearch lacks: Logstash reaches outside the search cluster, into databases, queues, and files, and pulls. Elasticsearch, by contrast, waits to be fed. Nothing inside Elasticsearch connects to your Postgres database, tails your log files, or polls your message queue. If data arrives in an index, something else brought it there.
A concrete pass makes the division obvious. Take a raw web-server log line: on its own it is one long string, useless for searching by status code or country. A Logstash pipeline reads the file (extract), runs the line through a grok filter to split it into timestamp, method, path, and status fields, adds a geoip lookup against the client IP, parses the timestamp into a real date, and only then sends the finished document to Elasticsearch (load). Every interesting thing that happened to that line happened before Elasticsearch saw it. What Elasticsearch then does with the document — indexing it so those fields can be searched and aggregated at volume — is a different talent, and whether that talent holds up at scale is its own question, covered in is Elasticsearch good for big data?
What about ingest pipelines? Elasticsearch’s built-in transforms
Ingest pipelines are the strongest argument on the other side, so they deserve a fair hearing. An ingest pipeline is a set of processors attached to Elasticsearch itself that transform a document just before it is indexed: set a field, uppercase a value, remove a field, parse a string, and so on. For light cleanup of documents that are already on their way into the cluster, they work well and avoid running a separate service.
Elastic’s own positioning is the clearest guide to their scope. Elastic’s blog describes the ingest node as offering “a subset of the functionality that Logstash supports,” aimed at situations where you do not require the full power of Logstash. That word — subset — is the honest summary. Ingest pipelines:
- Only see documents that are already arriving at Elasticsearch. They extract nothing from anywhere.
- Run inside the destination, at index time, so the transformation is coupled to the indexing process rather than managed as an independent pipeline.
- Handle per-document field manipulation well, but offer nothing like Logstash’s source connectors, routing, or multi-destination delivery.
Elastic’s documentation frames the division of labour across the whole stack in the same spirit: Beats and Elastic Agent can manipulate data at the edge, ingest pipelines manipulate it as it arrives, and Logstash manipulates it before sending, with the ability to route to multiple destinations. The two can also be combined — Logstash feeding an ingest pipeline is a supported pattern. None of that turns the destination into the ETL tool; it means the border between the pipeline and the store has some tooling on both sides of it.
Side by side: who does which job
| Capability | Elasticsearch | Logstash | ElasticDump |
|---|---|---|---|
| Connects to arbitrary sources (databases, queues, APIs, files) | No | Yes — 200+ plugins | No — Elasticsearch/OpenSearch, files, S3, CSV, stdio |
| Transformation layer | Ingest pipelines only (subset, at index time) | Yes — filters, enrichment, parsing | Per-document JavaScript via --transform |
| Delivers to multiple destinations | No — it is a destination | Yes | One input to one output per run |
| Scheduling / orchestration | No | Runs as a managed pipeline service | No — you run it when you need it |
| Primary job | Store, index, search, analyse | Move and transform data into destinations | Copy and back up Elasticsearch indices |
The table makes the pattern hard to miss: the further right a capability sits in the “pipeline” direction, the less Elasticsearch itself is involved. Elasticsearch anchors the storage end; Logstash owns the pipeline end; elasticdump occupies a narrow, specific lane described next. It is worth reading the table by what is absent as much as by what is present. The rows describe a relay — sources at one end, a destination at the other, transformation in the middle — and Elasticsearch appears in that relay exactly once, at the receiving end. Every earlier stage is staffed by something else, and a tool whose only position in the relay is the finish line is not the thing running the race.
Where ElasticDump honestly fits
Elasticdump is a utility for moving and saving indices from Elasticsearch and OpenSearch: you give it an input and an output, and each can be a cluster URL or a file (with S3, CSV, and standard input/output also supported). Looked at through ETL glasses, it performs something like the E and the L — but only between Elasticsearch-shaped endpoints. It reads documents out of an index and writes them into another index or a portable file. That is data movement, and useful data movement, but movement inside one ecosystem is not extraction from arbitrary sources.
The file it produces underlines the point. A dump is line-delimited JSON: each line is one complete document, the file as a whole deliberately is not a single JSON document, so dumps can be streamed, appended to, inspected, and loaded back later. That is the artefact of a copy tool — a faithful, portable snapshot of an index’s contents — not the output contract of a pipeline platform, which would be expected to land data in whatever shape and system the business uses, on a schedule, forever.
Its transformation story is similarly real but narrow. The --transform option runs a small piece of JavaScript against each document as it passes through, with the document available as a global doc object:
--transform 'doc._source["f2"] = doc._source.f1 * 2;'That is enough to compute one field from another, rename something, or anonymise sensitive values during a copy — the project ships an example transform for anonymising data on the fly, and transforms can also be loaded from a JavaScript module file and stacked, using more than one per run. Useful, certainly. But measure it against an ETL transform layer and the gap is plain: there are no enrichment lookups against other systems, no joins, no parsing frameworks, no routing of different documents to different places. Each document is adjusted in isolation, on its way from one Elasticsearch-shaped place to another. What elasticdump does not have is everything else that would make it an ETL platform: no database or queue connectors, no scheduling, no pipeline management. It runs when you run it, copies what you point it at, and stops.
That narrowness is a feature rather than an apology. When the job is a copy — staging refreshed from production, an index archived to a file, a cluster migration — the practical route is the one covered in how to copy data from one Elasticsearch cluster to another, and a general ETL platform would be heavier machinery than the task needs. And if you are still deciding what Elasticsearch itself is in your architecture, that question is taken up in is Elasticsearch a database? — the answer there explains a lot about why it is not an ETL tool either: in both cases, it is a specialist store with a few borrowed features, not a general-purpose platform for someone else’s job.
FAQ
Is Logstash an ETL tool?
Yes — it is the ETL component of the Elastic stack. Its inputs extract from many source types, its filters transform (grok parsing, geoip enrichment, field mutation, date handling), and its outputs load into Elasticsearch and other destinations, including several destinations at once.
Can ingest pipelines replace Logstash?
For light, per-document cleanup at index time, often yes — that is what they are for. Elastic itself describes them as a subset of Logstash’s functionality, so when you need source connectors, heavy parsing, enrichment, or routing to multiple destinations, you have outgrown them.
Can Logstash and ingest pipelines be used together?
Yes. Elastic’s own guidance treats them as composable rather than rivals: a Logstash pipeline can do the heavy extraction and transformation work and deliver documents into an ingest pipeline for a final, index-specific adjustment — or the edge tooling can handle part of the shaping before either of them sees the data.
Is elasticdump an ETL tool?
No. It is a data-movement utility for Elasticsearch and OpenSearch indices. Its --transform option applies small per-document JavaScript changes during a copy, but it has no connectors to external systems, no scheduling, and no pipeline management.
Can Elasticsearch pull data from my database by itself?
No. Elasticsearch has no source connectors and does not initiate extraction. Something — Logstash, Beats, Elastic Agent, or your own code — has to read the source and send documents to Elasticsearch for indexing.
Where does transformation happen in a typical Elastic setup?
In three possible places, per Elastic’s documentation: at the edge with Beats or Agent, in a Logstash pipeline before delivery, or inside Elasticsearch via ingest pipelines as documents arrive. Teams commonly combine them rather than picking exactly one.
Do I need Logstash just to move data between two Elasticsearch clusters?
No. Logstash earns its place when a source or destination lives outside Elasticsearch — a database, a queue, log files, another system entirely. A cluster-to-cluster copy is a narrower job with no external extraction involved, and it belongs to a movement utility, not a pipeline platform.
Conclusion
Elasticdump exists to move Elasticsearch data, and the question this article started with dissolves once the jobs are labelled properly: Elasticsearch stores, indexes, and searches; Logstash extracts, transforms, and loads; ingest pipelines do light transformation inside the store at index time; and elasticdump copies indices between clusters and files, with small per-document adjustments available along the way. Calling Elasticsearch an ETL tool credits the destination with the pipeline’s work. Keep each tool on the job it was built for — and reach for the ETL tool when the source is a database, a queue, or a log file, not when it is another index.