how to copy one index documents to other index in elasticsearch?

Question

I created a new index with mappings. 500 000 documents are stored in it.

I want to change the mapping of the index, but it is not possible in elastic search.so I created another index with new new mappings and now I am trying to copy the documents from old index to new index.

I am using scan and scroll type to retrieve the documents from old index and copying to new index.for copying its taking more time and the system is slowing down.

Below is the code that I am using.

$client= app('elastic_search');
    $params = [
        "search_type" => "scan",    // use search_type=scan
        "scroll" => "30s",          // how long between scroll requests. should be small!
        "size" => 500000,               // how many results *per shard* you want back
        "index" => "admin_logs422",
        "body" => [
            "query" => [
                "match_all" => []
            ]
        ]
    ];

    $docs = $client->search($params);   // Execute the search

    $scroll_id = $docs['_scroll_id'];   


    while (\true) {

        // Execute a Scroll request
        $response = $client->scroll([
                "scroll_id" => $scroll_id,  //...using our previously obtained _scroll_id
                "scroll" => "500s"           // and the same timeout window
            ]
        );

        if (count($response['hits']['hits']) > 0) {
            foreach($response['hits']['hits'] as $s)
            {

                $params =
                    [
                        'index' => 'admin_logs421',
                        'type' => 'admin_type421',
                        'id'=> $s['_id'],
                        'client' => [
                            'ignore' => [400, 404],
                            'verbose' => true,
                            'timeout' => 10,
                            'connect_timeout' => 10
                        ],
                        'body' => $s['_source']
                    ];

                $response = app('elastic_search')->create($params);

            }


            $scroll_id = $response['_scroll_id'];
        } else {
            // No results, scroll cursor is empty.  You've exported all the data
            return response("completed");
        }
    }

score 6 · Accepted Answer · edited May 23 '17 at 12:10

6

You should not have to code stuff like that. There are some excellent tools around to do it for you.

Just look at Taskrabbit's elasticdump utility which does exactly what you want.

elasticdump \
  --input=http://localhost:9200/source_index \
  --output=http://localhost:9200/target_index \
  --type=data

Or you can also very easily leverage Logstash as is shown in this other answer

Finally, since you're using Python, you can also use the elasticsearch-py reindex utility

edited May 23 '17 at 12:10

Community

1
1

answered Jan 21 '16 at 11:29

Val

207,596
13
358
360

Not working.after copying data from one index to another,no data found in new index – Veerendra Borra Jan 23 '16 at 06:22
You've replaced `source_index` by `admin_logs422` and `target_index` by `admin_logs421`, right? – Val Jan 23 '16 at 07:25
i have a index admin_logs221 and type admin_type221 with mappings and 30000 documents in it. i created another index with one change in mapping as admin_logs mapping.iam using docker command to copy data – Veerendra Borra Jan 25 '16 at 08:53
docker run --rm -ti elasticdump --input=http://localhost:9200/admin_logs421 --output=http://localhost:9200/admin_logs422 --type=data --limit=10000 --bulk=true --scrollTime=30s – Veerendra Borra Jan 25 '16 at 08:58
starting dump got 10000 objects from source elasticsearch (offset: 0) sent 10000 objects to destination elasticsearch, wrote 10000 got 570 objects from source elasticsearch (offset: 10000) sent 570 objects to destination elasticsearch, wrote 570 got 0 objects from source elasticsearch (offset: 10570) sent 0 objects to destination elasticsearch, wrote 0 Total Writes: 10570 dump complete – Veerendra Borra Jan 25 '16 at 09:00
no data is found in the indices..documents didn't transferred. – Veerendra Borra Jan 25 '16 at 09:02
And you're sure you have mapped the port 9200 with `-p 9200:9200`? – Val Jan 25 '16 at 09:02
where i have to map the port -p 9200:9200 – Veerendra Borra Jan 25 '16 at 09:09

Pandiyan Cool · Answer 2 · 2016-01-21T11:40:41.567

0

you can try using the some existing reindexing plugins

https://github.com/codelibs/elasticsearch-reindexing

by sending the following request

localhost:9200/{fromindex}/{fromtype}/_reindex/{toindex}/{totype}

edited Jan 21 '16 at 11:40

answered Jan 21 '16 at 11:15

Pandiyan Cool

6,381
8
51
87

getting error when copying data from one index to another – Veerendra Borra Jan 23 '16 at 06:23
can you please paste your query and error you have got here – Pandiyan Cool Jan 25 '16 at 04:02
http://localhost:9200/admin_logs421/admin_type421/_reindex/admin_logs422/admin_type422 No handler found for uri [/admin_logs421/admin_type421/_reindex/admin_logs422/admin_type422] and method [GET] – Veerendra Borra Jan 25 '16 at 08:47
1

You need to [install the plugin](https://github.com/codelibs/elasticsearch-reindexing) first. – Val Jan 25 '16 at 08:50

how to copy one index documents to other index in elasticsearch?

2 Answers2

Linked

Related