Elasticdump exists because this question is harder than it looks. Elasticsearch is built to answer questions about an index — find these documents, count those, aggregate the rest — and it is quietly resistant to being emptied out. Ask it for the contents of an index in the obvious way and it hands back ten documents and considers the job done, whether the index holds ten documents or ten million.
That resistance is deliberate, and understanding it is most of the answer. Elasticsearch protects itself with pagination limits, so getting everything out means choosing a method designed for walking an entire index rather than searching it. This article climbs the ladder of options in order: the default search and why it stops at ten, from/size and its 10,000-document wall, the legacy scroll API, the currently recommended search_after with a point in time, and finally the route most people actually want — letting a tool built for the job do the walking, then verifying that what arrived matches what left.
The trap: a search returns 10 documents
The most common version of this problem starts with a perfectly reasonable request:
GET /my-index/_search
{
"query": { "match_all": {} }
}
The response is a success. It contains documents. It also contains only the top 10 hits, because 10 is the default page size for a search — and unless you read the total value in the response, nothing about the output warns you that you are looking at a sample. A large share of “I exported my index but the file looks small” reports turn out to be exactly this: not a failed export, just page one of many.
Method 1: from/size — fine up to 10,000
The familiar pagination parameters work exactly as expected at small scale. size sets how many hits come back and from sets where the page starts, so "from": 9900, "size": 100 returns hits 9,901 to 10,000. For an index with a few hundred or a few thousand documents, raising size — or paging through with from — is genuinely all you need.
Then you hit the wall. The combined window is capped by the index setting index.max_result_window, which defaults to 10,000. Ask for a page that ends beyond it and the request is rejected. Elastic is explicit about why the safeguard exists: search requests take heap memory and time proportional to from + size. To serve page 900 of a search, every shard has to collect and sort its candidates for the entire window, so deep from/size paging gets more expensive with every page you turn.
You can raise max_result_window, and plenty of old forum answers suggest exactly that. Elastic’s own guidance points the other way: raising it trades a safeguard for heap pressure on every shard, and the documented recommendation is to use search_after for deep pagination instead. Raising the window to 100,000 to export an index of two million documents also still leaves you 1.9 million documents short. It is a workaround for moderate result sets, not an extraction method.
One clarification, because it causes real confusion: the 10,000 cap applies to the from/size result window only. It is not a limit on how many documents the remaining methods in this article can return.
Before leaving this method, one habit is worth forming: read the total in any search response. It is the ground truth that every export gets measured against — the number your pages, batches and output files have to add up to at the end. Every method below is, in the end, a way of making the delivered count meet that number.
Method 2: the scroll API — the legacy workhorse
Scrolling was the original answer to “give me everything.” You open a scroll with an initial search, and Elasticsearch keeps a search context alive — effectively a point-in-time view of the index — while you call the scroll endpoint repeatedly to pull batches until they come back empty:
POST /my-index/_search?scroll=1m
{ "size": 1000, "query": { "match_all": {} } }
POST /_search/scroll
{ "scroll": "1m", "scroll_id": "..." }
The guarantees are good. The documents you get are the documents as they were when the scroll opened: things indexed or changed afterwards do not leak into the result and shift your pages around mid-export. And there is no 10,000 cap on a scroll — it can walk an index of any size.
The costs are why scroll is no longer the recommended default. A scroll context holds state on the cluster for its entire lifetime, and a long extraction over a big index holds it for a long time. Elastic’s documentation is blunt that scroll is not recommended for real-time user requests, and current guidance recommends search_after with a point in time for nearly all pagination, reserving scroll for bulk export and legacy tooling. For very large jobs there is sliced scrolling — splitting one scroll into parallel slices, up to a default maximum of 1024 slices per scroll (index.max_slices_per_scroll) — but slicing multiplies the held state rather than removing the caveat.
Scroll still matters in practice for one reason: it is what a generation of export tooling, including ElasticDump’s standard reading mode, was built on. It works, it is battle-tested for bulk export, and on a modern cluster it is best treated as the compatibility option rather than the first choice.
There is also a practical failure mode to plan around. The context only survives between requests for the keep_alive window you set, renewed with each call. A consumer that stalls — a slow disk at the receiving end, a paused pipeline — for longer than that window comes back to a scroll that no longer exists, and the export restarts from zero. On a small index that is an annoyance. On a hundred-million-document index, discovered four hours in, it is the reason people move to the methods below, or to tooling with resuming built in.
Method 3: search_after with a point in time — the recommended route
The method Elastic currently documents for deep pagination combines two pieces. A point in time (PIT) freezes the view of the index; search_after then pages through that view by using the last hit of each page as the cursor for the next, instead of counting from the beginning:
POST /my-index/_pit?keep_alive=1m
POST /_search
{
"size": 1000,
"pit": { "id": "...", "keep_alive": "1m" },
"sort": [ { "created_at": "asc" }, { "_id": "asc" } ],
"search_after": [ 1699999999000, "abc123" ]
}
Why this shape works:
- No deep-paging cost. Each request only needs the documents after the cursor, not the whole window before them, so page 900 costs about the same as page 2.
- A stable view. The requests run against a PIT, which exists precisely to preserve the current index state over your searches — refreshes and newly indexed documents between pages cannot reorder the results.
- A deterministic order. The sort must include a unique tiebreaker field (commonly the document
_id) so that “the last hit” is an exact position rather than an approximate one. Without it, documents sharing the same sort value can be skipped or repeated.
The mechanics are a simple loop: take the sort values from the final hit of each response, feed them back as search_after, and stop when a page comes back empty. It is more code than from/size, and it is the method Elastic steers you to for anything beyond shallow paging — exports included.
Method 4: ElasticDump — let a tool do the walking
Every method above is something you implement and babysit. For most people asking this question, the actual goal is a file, a backup, or a copy in another cluster — and that is the job ElasticDump was built for. It implements the walking itself: under the hood it reads with the scroll API described above, and on current Elasticsearch versions it can use the modern route instead, via its --searchAfter option (which requires Elasticsearch 7.17.0 or later) and --pit support.
What you hand it is an input and an output. The input is your index URL; the output can be another cluster’s index, a local file, an S3 location, a CSV file, or $ for stdout. The homepage carries the copy-paste recipes — full backup to a file, CSV export, gzip-compressed output — so they are not repeated here; the shape is always the same:
elasticdump \ --input=http://localhost:9200/my-index \ --output=/backups/my-index.json \ --type=data
Two of its behaviours matter specifically for “all data” jobs, because they are the parts the raw APIs leave to you:
- Batching runs to the end. ElasticDump pulls documents in batches — 100 per batch by default — and keeps going until the index is exhausted. There is no page-size trap to fall into and no loop for you to write.
- Interrupted runs can resume. If a long export dies halfway,
--scrollIdpicks up from the last scroll ID while the scroll context is still alive, and--offsetrestarts from a known written line in the output file — with the honest caveat from the documentation that, since no sorting is applied, skipped rows cannot be guaranteed to be the already-written ones.
If the destination is another cluster rather than a file, the workflow widens to copying structure before data, in the right order — that full process is covered in how to copy data from one Elasticsearch cluster to another.
One format note, because it changes what “all data” means in the output file. The standard dump format is line-delimited JSON: the file as a whole is not valid JSON, but each line is one complete document, which is what makes dump files easy to stream, append to and count. CSV output is the exception to keep in mind — the CSV transport flattens nested document data down to a single level, in a documented and deliberately opinionated way, and Elasticsearch documents are nested by nature. For a faithful copy, JSON lines are the format to keep; CSV is for handing data to tools that need rows and columns.
The methods side by side
| Method | Practical limit | Consistent view? | Best for |
|---|---|---|---|
| Plain search | 10 hits by default | No | Testing a query |
| from/size | 10,000-hit result window by default | No | Small indices, shallow pages |
| Scroll API | No count cap; context lifetime is the cost | Yes — frozen when opened | Legacy tooling, bulk export |
| search_after + PIT | No count cap | Yes — PIT preserves state | Programmatic extraction, deep paging |
| ElasticDump | No count cap; resumable | Yes — scroll or PIT underneath | Files, backups, cluster copies |
Verify you actually got everything
Whichever method you use, finish with a count comparison. Count the documents in the source index — the _count API, or _cat/indices — and count what arrived: documents in the destination index, or lines in the export file, since a dump file holds one document per line. If the numbers disagree, the export is not done, whatever the tooling reported.
This step matters even more with ElasticDump than with hand-rolled code, for one documented reason: its --ignore-es-write-errors option defaults to true, so write errors during a load do not stop the run. A completed run is evidence that the tool finished, not that every document landed. When a load does come up short or fails outright, the common ElasticDump errors and fixes walks through the usual causes — connection failures, scroll problems and mapping rejections among them.
When the right answer is a snapshot instead
Sometimes “get all the data” really means “make sure we can get this index back.” If the goal is backup or disaster recovery rather than documents you can open, inspect or load somewhere else, a snapshot is the better tool: it preserves the index in Elasticsearch’s own repository format and restores it into a compatible cluster. What it does not give you is usable data — a snapshot is not a file of documents another tool can read. The trade-offs between the two approaches are laid out in ElasticDump versus snapshot and restore. If a person needs to read the data, extract it; if a cluster needs to recover it, snapshot it.
FAQ
How do I export an Elasticsearch index to a file?
Use ElasticDump with the index as --input and a file path as --output (the recipes are on the homepage), or page through the index with search_after and a point in time and write the results out yourself. Either way, do not rely on a plain search — it returns 10 documents by default.
Why did my export only contain 10 documents?
Because a search returns the top 10 hits unless you say otherwise. It is the single most common cause of “incomplete” exports. Set a page size and use one of the walking methods in this article, and check the total in the first response so you know the real target number before you start.
Can I just set size to 1,000,000?
No. The from/size window is capped by index.max_result_window, which defaults to 10,000. Requests beyond the window are rejected, and raising the setting is discouraged — it costs heap memory and time proportional to from + size on every shard.
Is scroll limited to 10,000 documents?
No. The 10,000 figure is the from/size result window. A scroll can return an entire index; its real constraints are the state it holds on the cluster and the lifetime of its context, which is why Elastic now recommends search_after with a point in time for most pagination.
Which method should I use in code?
search_after with a point in time. Open a PIT, sort by a field plus a unique tiebreaker, and feed each page’s last sort values back as search_after until a page returns empty. It has no count cap, no deep-paging cost, and a stable view of the index while you work.
Can I extract the data while the index is still being written to?
Yes, but pick a method with a frozen view. A scroll and a point in time both show you the index as it was when they opened, so documents written during the export simply are not in it — which is normally what you want from an export. Plain from/size paging has no such guarantee: on a busy index its pages can shift underneath you between requests, repeating some documents and skipping others.
Conclusion
Elasticdump was built on a simple observation: Elasticsearch will happily answer questions about an index all day, but handing over the whole index is a job it makes you ask for properly. The default search gives you 10 documents, from/size gives you 10,000, and past that you are in the territory of scroll, search_after with a point in time, or a tool that wraps those APIs for you. Pick the method that matches the job — search_after and PIT for code, ElasticDump for files, backups and cluster copies, a snapshot for disaster recovery — and always close the loop by counting what arrived. “All the data” is a claim you verify, not one a successful-looking response makes for you.