C# HttpClient.PostAsJsonAsync() fails, even though the exact same request is accepted when made through PostMan

Question

I have found similar questions both here as well as on the Elastic discussion forum, but unfortunately none of the answers helped.

I am currently using ElasticSearch 7.0.

I want to make a bulk request to my ElasticSearch server. My JSON file contains information that looks something like this:

{ "index": { "_index": "website", "_id": "link1" }}
{ "label":    "Link1" }

Each line is terminated by an LF line break, and there is also an additional LF line break at the end of the document.

In C#, here is how I make a POST request for my bulk data:

HttpResponseMessage response = await httpClient.PostAsJsonAsync($"http://127.0.0.1:9200/website/_bulk", jsonDocumentContents);

And yet I keep seeing this error message:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"},"status":400}

How can I fix this error?

UPDATE:

A short description of how I read the JSON document contents into the jsonDocumentContents variable: The JSON document was stored inside a zipped folder, so retrieving it requires unzipping:

ZipArchive archive = new ZipArchive(zippedFolderStream);
foreach (ZipArchiveEntry entry in archive.Entries)
{
    string jsonDocumentContents = new StreamReader(entry.Open()).ReadToEnd();
    HttpResponseMessage response = await httpClient.PostAsJsonAsync($"http://127.0.0.1:9200/website/_bulk", jsonDocumentContents);
    Console.WriteLine(await response.Content.ReadAsStringAsync());
}

UPDATE:

I just made a bulk request with the exact same contents using PostMan, and the request was successful. However, the error message persists when I make the same bulk request in C# using httpClient.PostAsJsonAsync(...).

The question is probably how do you read the file content into the `jsonDocumentContents` variable? — Val, Apr 30 '19 at 12:47
@Val Thank you for your comment. I have updated my post to include information that answers your question. — M.Y. Babt, Apr 30 '19 at 12:53
Possible duplicate of [Bulk request throws error in elasticsearch 6.1.1](https://stackoverflow.com/questions/48579980/bulk-request-throws-error-in-elasticsearch-6-1-1) — stop-cran, Apr 30 '19 at 12:56
@stop-cran There is already an empty line at the end of my file. — M.Y. Babt, Apr 30 '19 at 12:57

M.Y. Babt · Accepted Answer · 2019-04-30T17:20:40.427

I got it working by changing my code to the following:

ZipArchive archive = new ZipArchive(zippedFolderStream);
foreach (ZipArchiveEntry entry in archive.Entries)
{
    string jsonDocumentContents = new StreamReader(entry.Open()).ReadToEnd();
    StringContent content = new StringContent(jsonDocumentContents, Encoding.ASCII, mediaType: "application/json");
    HttpResponseMessage response = await httpClient.PostAsync($"http://127.0.0.1:9200/website/_bulk", content);
    Console.WriteLine(await response.Content.ReadAsStringAsync());
}

Notice that I am using HttpClient.PostAsync() instead of HttpClient.PostAsJsonAsync(), with a StringContent instance that specifies "application/json" as its media type.

I looked into the source code for HttpClient, and noticed that a new instance of JsonMediaTypeFormatter is created every time HttpClient.PostAsJsonAsync is called.

Since my POST requests are successful when I make them through PostMan, the issue must be caused by how PostAsJsonAsync() is implemented. I suspect, but have not verified, that the problem is due to the default properties in the JsonMediaTypeFormatter class.

To circumvent the problem I decided to use Http.PostAsync() with a correctly-configured StringContent instance.

Lo and behold, I can now send bulk requests to my ElasticSearch server using C#.

`PostAsJsonAsync` will *serialize whatever you give it to JSON*. So if you pass in a string like `{ first:0 }\n{ second:1 }`, it will serialize the string to JSON as such: `"{ first:0 }\\n{ second:1 }"` and then send it. When ES gets it and deserializes it, it only would see a single JSON string, not a series of JSON objects. — Stephen Cleary, Apr 30 '19 at 14:35
pretty sure it was related to some C# method doing some smart-ass conversion. glad you figured it out — Val, Apr 30 '19 at 15:04
Man! Wish I found this post earlier. WTF MS 4 hours wasted cause of this PostAsJsonAsync bs!!! — ihor.eth, Jan 19 '22 at 05:16

C# HttpClient.PostAsJsonAsync() fails, even though the exact same request is accepted when made through PostMan

1 Answers1