Elasticsearch: Bulk request throws error in Elasticsearch 6.1.1

Question

I recently upgraded to Elasticsearch version 6.1.1 and now I can't bulk index documents from a JSON file. When I do it inline, it works fine. Here are the contents of the document:

{"index" : {}}
{"name": "Carlson Barnes", "age": 34}
{"index":{}}
{"name": "Sheppard Stein","age": 39}
{"index":{}}
{"name": "Nixon Singleton","age": 36}
{"index":{}}
{"name": "Sharron Sosa","age": 33}
{"index":{}}
{"name": "Kendra Cabrera","age": 24}
{"index":{}}
{"name": "Young Robinson","age": 20}

When I run this command,

curl -XPUT 'localhost:9200/subscribers/ppl/_bulk?pretty' -H 'Content-Type: application/json' -d @customers_full.json

I get this error:

"error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "The bulk request must be terminated by a newline [\n]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "The bulk request must be terminated by a newline [\n]"
  },
  "status" : 400

It works fine if I send the data inline and in Elasticsearch 5.x. I tried adding newlines as well as the newline character to the end of the file. Doesn't seem to work.

score 72 · Accepted Answer · edited Jan 14 '21 at 07:23

72

Add empty line at the end of the JSON file and save the file and then try to run the below command

curl -XPOST localhost:9200/subscribers/ppl/_bulk?pretty --data-binary @customers_full.json -H 'Content-Type: application/json'

edited Jan 14 '21 at 07:23

Sabito stands with Ukraine

4,271
8
34
56

answered Mar 16 '18 at 22:12

Sathishkumar Rakkiyasamy

3,509
2
30
34

1

it was so simple, I was trying to add a new line character to command itself :| ..whereas it clearly says '*request* must be terminated by a new line' (y) – Vikash Mar 25 '19 at 09:44
1

I added the new line but still the issue is showing – TheEhsanSarshar Sep 28 '20 at 17:44
1

It worked as I added a newline to the json + used `--data-binary` parameter instead of `-d` or `--data`. Adding a newline alone doesn't help. – maximus Oct 05 '20 at 04:35
Where to create@customers_full.json file? – NightOwl Mar 23 '22 at 17:23
You can create anywhere in the file system. But you need to execute the above command from the same location of the JSON file. – Sathishkumar Rakkiyasamy Mar 24 '22 at 04:36

score 25 · Answer 2 · edited Nov 07 '19 at 08:49

25

As the document says: use the --data-binary flag instead of plain -d

-d doesn’t preserve newlines and doesn't format the json.

I faced this problem because of JSON formatting.

edited Nov 07 '19 at 08:49

avp

2,892
28
34

answered Jun 01 '18 at 20:28

Android14

1,045
1
11
18

3

This answer REALLY needs to upvoted to go to the top. `--data-binary` is indeed different from `-d` even on Mac. I had thought that binary mode ONLY made a difference in windows platform! – pktCoder Mar 02 '19 at 17:31

score 8 · Answer 3 · answered Feb 02 '18 at 09:53

8

The error is pretty clear:

The bulk request must be terminated by a newline [\n]

So you simply need to add a newline at the end of your customers_full.json file and you'll be ok.

answered Feb 02 '18 at 09:53

Val

207,596
13
358
360

5

I did that. I added a newline. I tried adding the newline character, still doesn't work. – Judy T Raj Feb 02 '18 at 09:54
I tried the same thing against ES 6.1.3 and it worked perfectly – Val Feb 02 '18 at 10:04
5

Instead of `-d` you should try to use `--data-binary` – Val Feb 02 '18 at 10:41
It's something like this https://gist.github.com/Katniss394/f5b44ae76df85246118b3e030e6f8680 – Judy T Raj Feb 28 '18 at 07:54
well, somehow an explicit new line is missing at the end of your file, we should see a complete empty line as the last line of your document – Val Feb 28 '18 at 11:28

score 7 · Answer 4 · answered May 30 '18 at 20:45

7

I ran into the same issue and spent hours adding and removing newlines before somebody pointed out I mis-typed the file name... So note that curl will throw the same error if the file is not actually present, making this super-confusing.

answered May 30 '18 at 20:45

Raya Fratkina

71
1
1

Promise Preston · Answer 5 · 2021-01-14T11:26:58.437

I had a similar issue when working with Elasticsearch 7.3.

Here's how I solved it.

Locate the .json file, say products.json file.
Double click to open the .json file in your text editor.
Scroll to the end of the .json file and then press the ENTER key on your keyboard.
Close the .json file. This will create a new line at the end of .json file.
Go back to your terminal and run the command below.

N/B: For the command below, the .json file name is products.json which I am importing to http://localhost:9200/ecommerce/product

curl -H "Content-type: application/json" -XPOST "http://localhost:9200/ecommerce/product/_bulk?pretty" --data-binary "@products.json"

That's all.

I hope this helps

score 4 · Answer 6 · answered Apr 15 '19 at 10:55

4

For anyone using postman to make requests to ElasticSearch

Just press enter to create an empty new line!

And voila, problem solved

answered Apr 15 '19 at 10:55

bnahayo

121
1
1

score 4 · Answer 7 · edited Jul 10 '19 at 11:54

4

This worked for me:

curl -H "Content-Type: application/x-ndjson" -XPOST "localhost:9200/bank/_bulk?pretty&refresh" --data-binary "@C:\Program Files\Elastic\Elasticsearch\7.2.0\accounts.json"

edited Jul 10 '19 at 11:54

Shayki Abramczyk

36,824
16
89
114

answered Jul 10 '19 at 11:18

Rajat Goel

143
1
8

score 2 · Answer 8 · answered Feb 07 '18 at 07:47

2

you just need to open json file and then go to the end of the file ( Ctrl+end) and then please Enter to break a new line.

answered Feb 07 '18 at 07:47

user6589750

21
2

wow! u just saved my life!!! – NanoNova Apr 26 '23 at 07:55

score 2 · Answer 9 · edited Feb 08 '20 at 14:57

2

Press Enter end of the line inside the JSON file and run the command again.

curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"generated.json"

edited Feb 08 '20 at 14:57

David Buck

3,752
35
31
35

answered Oct 09 '18 at 19:22

sourabh kumar verma

21
3

score 2 · Answer 10 · answered Jan 17 '20 at 16:22

I had the same problem running on Windows 10, using ElasticSearch 7.5.1.

I tried all the answers; none of them worked. I was certain I had a newline at the end of my file.

To get it to work, I had to ensure the file I was uploading was using UNIX end-of-line characters (0A only, no 0D), and also the encoding had to be UTF-8.

Using Notepad++, you can edit the metadata of the file.

Finally some good news:

TheEhsanSarshar · Answer 11 · 2022-04-16T16:53:45.603

2

I just forgot to add an @ symbol before file name like this

--data-binary "@products.json"

edited Apr 16 '22 at 16:53

answered Sep 28 '20 at 17:49

TheEhsanSarshar

2,677
22
41

score 1 · Answer 12 · edited Jan 14 '21 at 07:27

1

You need to use --data-binary instead of -d in your curl request. Please see: Bulk API

edited Jan 14 '21 at 07:27

Sabito stands with Ukraine

4,271
8
34
56

answered Mar 16 '18 at 20:54

Karim

11
1

score 1 · Answer 13 · edited Jan 14 '21 at 07:25

1

For me, the issue was only due to the wrong file name. I have used customer_full.json in command whereas the file was named customer_full in my file system (without the extension).

So in my case,this command worked for me:

curl -H "Content-Type: application/x-ndjson" -XPOST 'http://localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"customer_full"

edited Jan 14 '21 at 07:25

Sabito stands with Ukraine

4,271
8
34
56

answered Feb 08 '20 at 14:26

Shubham Paldewar

51
4

score 1 · Answer 14 · edited Jan 14 '21 at 07:26

I faced a similar issue on windows using elastic 7.9.1 When I used below CURL command.

curl -s -H "Content-Type: application/json" -XPOST localhost:9200/accounts/docs/_bulk?filter_path=items.*.error --data-binary  "@textoutES.json"  >> erroredAtES.json"

I tried to manually add Newline at the end of the file but did not work.

I have created my JSON by extracting data from MySQL database like below to make sure my records should end with LINE FEED and CARRIAGE RETURN.

Then, it is working for me:

SELECT CONCAT('{"index":{"_id":"',id,'"}}\r\n',request_data,'\r\n') reqestData FROM cards

More importantly you End-of-File should have a carriage-return and Line-Feed (CRLF)if you are using windows. Also if any line in JSON contains a CR but no LF then you will get parsing exception Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@2d5ed2ca

Windows CRLF and EOF

score 1 · Answer 15 · edited Jan 14 '21 at 07:25

1

I was struggling with this for a hot minute. Mine was caused by a space in my curl request between the --data and the -binary and gave the same error - must end with new line [\\n]}.

So double-check that in the curl req it's --data-binary not --data - binary

edited Jan 14 '21 at 07:25

Sabito stands with Ukraine

4,271
8
34
56

answered Oct 15 '20 at 17:56

Brian Loveless

13
5

score 0 · Answer 16 · edited Feb 08 '20 at 14:57

0

This worked in my local set-up.

curl -H "Content-type:application/json" -XPOST "http://localhost:9200/customer/personal/_bulk?pretty" --data-binary @"generated.json"

edited Feb 08 '20 at 14:57

David Buck

3,752
35
31
35

answered Mar 19 '19 at 21:28

meol

1,027
11
7

score 0 · Answer 17 · answered Dec 06 '21 at 11:42

How do you do that if you are not using a data-file? I am having the issue but not sending data from a file.

const data1 = {
    "amount" : "100",
    "@timestamp" : `${UTC_timestamp}`,
    "transaction_attributes" : {
    "channel" : "channel-foobarbaz",
    "session_id" : "session-1234",
    "information" : "iinformation-foobarbaznformation-foobarbaz"
    },
    "currency" : {
    "currency_description" : "my currency description",
    },
    "external_timestamp" : "2021-12-03T11:22:55.206229500Z" };

  
// execute a post
let res = http.post(url,JSON.stringify(data1),params);

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). — Community, Dec 06 '21 at 13:33

score 0 · Answer 18 · answered Nov 23 '22 at 23:09

0

A few things to check:

The file ends with new line (\n).
The new line is using Unix eol (LF) and not mac or windows eol.
When specifying the file name in the curl command, make sure "@" was added before the file name.

answered Nov 23 '22 at 23:09

Yaki Klein

3,978
3
37
34

Garry Kasprov · Answer 19 · 2023-03-21T11:17:44.063

0

When I use /Users/myName/file.json the question show, but when I use @/Users/myName/file.json it works.

edited Mar 21 '23 at 11:17

answered Feb 24 '23 at 06:44

Garry Kasprov

1
1

Elasticsearch: Bulk request throws error in Elasticsearch 6.1.1

19 Answers19

Linked