Error when trying to add index and document in Elasticsearch via Python requests.post()

Question

I'm trying to add index and document to Elasticsearch by requests.post() method:

elastic_url = 'http://localhost:9200/'
elastic_headers = {'Content-Type': 'application/json'}
test_json = json.dumps({"id": 1, "foo": "bar"})

requests.post(elastic_url + 'test/_doc/1', 
              json=test_json, 
              headers=elastic_headers)

Got the following error as response:

{'error': {'root_cause': [{'type': 'mapper_parsing_exception',
           'reason': 'failed to parse'}],
 'type': 'mapper_parsing_exception',
 'reason': 'failed to parse',
 'caused_by': {'type': 'not_x_content_exception',
               'reason': 'Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes'}},
 'status': 400}

When doing the same in terminal via curl:

curl -X POST -H 'Content-Type: application/json' 'http://localhost:9200/test/_doc/1' -d '{"id": 1, "foo": "bar"}'

the data being written successfully. Cannot figure out what is the reason, please help me.

Iris D · Answer 1 · 2021-06-07T01:47:39.610

When you make the requests post you dumped the json again, when all you had to do was send test_json.

See if this works:

requests.post(elastic_url + 'test/_doc/1', 
              json=test_json, 
              headers=headers)

That might fix the mapping error! If you still receive the mapping parsing exception like below, either update your mapping or check the data you're submitting.

'caused_by': {'type': 'not_x_content_exception',
               'reason': 'Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes'}}

Mapping example:

# mapping tells es what kind of data each field contains

mapping = {
  "mappings": {
      "properties": {
          "id": {
            "type": "long"
          },
          "foo": {
            "type": "text"
          }
         }
        }
       }

Do you need to use requests for this post?

If not, may I suggest the python elasticsearch client? You can do it like below. I often have to create an index and mapping and upload documents to elasticsearch and this is how I do it. (note - i wrap all these up into easy to use functions but figured I'd just show you the guts!)

from elasticsearch import Elasticsearch
from elasticsearch import helpers

# bulk generator function with the es required formatting for successful bulk upload. 
# the required dict is the index and source line below 
def bulk_gen(json_data):

    bulk_action = ({"_index": es_index,"_source": j,} for j in json_data if json_data is not None)
    
    for b in bulk_action:
        yield b
        print(f'processed {es_index}')

# connect to elasticsearch 
es = Elasticsearch(cloud_id=cloud_id,
                   http_auth=('username', 'password')
# name index 
es_index = 'index name here'

# create index, add mapping
es.indices.create(index=es_index, body=mapping, ignore=400)

# process your json data 
json_data = json.loads({"id": 1, "foo": "bar"})

# finally, let's upload everything! 
# we've already created an index and mapping and processed the json data into bulk upload format, now we let 'er rip       

try: 
    helpers.bulk(es, bulk_gen(json_data))
    print('Imported resource succesfully')
except Exception as e: 
    print(f'there was an error: {e}')

Hope this helps. I struggled and then started using the es python client and things got a lot easier.

oh.. dumping `test_json` second time is just a typo, in original code I make it only once and getting the error anyway, sorry for that. About python elasticsearch client: I want to keep the script as simple as possible and do not include any additional dependencies. — Kaderma, Jun 07 '21 at 01:54
Hmm, in that case I would go back to your mapping and see what the issue is there, since it's throwing a mapping parsing exception. This answer says that "lEasticSearch requires the outermost document to be a JSON object, not a JSON array or scalar value" and suggests adding curly brackets around the data object. https://stackoverflow.com/questions/35213472/elasticsearch-error-while-sending-data — Iris D, Jun 07 '21 at 02:09
I looked at my old code for a successfully executed POST request to elastcsearch and this worked: `first_data = json.load(infile)` `json_data = {'objects': first_data['objects']}` — Iris D, Jun 07 '21 at 02:29

Error when trying to add index and document in Elasticsearch via Python requests.post()

1 Answers1