1

I am trying to follow this answer, but I am getting error in the actions parameter of bulk method. I am able to use next to generate array of json objects but when I pass it to helpers.bulk I get error.

This is my code:

from elasticsearch import Elasticsearch, helpers
import sys, json
import os

es = Elasticsearch("localhost:9200")
filename = "path_to_file.json"
def load_json(filename):
#     " Use a generator, no need to load all in memory"
print(filename)
with open(filename,'r') as open_file:
    yield json.load(open_file)`

helpers.bulk(es, load_json(filename), index='cllimaster_test', doc_type='clli_data_test')

Error:

enter image description here

Murtaza Haji
  • 1,093
  • 1
  • 13
  • 32

1 Answers1

1

This python (not ES) error would occur if .pop() had been called on a list instead of a dict. I haven't seen your json file but here's your linted code which works perfectly fine:

from elasticsearch import Elasticsearch, helpers
import sys
import json
import os

es = Elasticsearch("localhost:9200")
filename = "path_to_file.json"


def load_json(filename):
    with open(filename, 'r') as open_file:
        yield json.load(open_file)


helpers.bulk(es, load_json(filename), index='cllimaster_test',
             doc_type='clli_data_test')

print(es.count(index='cllimaster_test'))
Joe - GMapsBook.com
  • 15,787
  • 4
  • 23
  • 68
  • Thanks, turns out my json was returning a list, fixed it by adding key at the beginning of array of json objects. I have new issue of read timeout now, i might post new question if not able to figure that out. Please be on the lookout :) – Murtaza Haji Apr 07 '20 at 23:11
  • My json file: `{rows:[{obj1},{obj2},{obj3}]}`. Kibana is considering `rows` as field and storing my objects as nested field. Thats why I am getting read timeout coz its considering the whole file as single object within rows. – Murtaza Haji Apr 08 '20 at 02:23
  • 1
    extract the rows like so `yield json.load(open_file)['rows']`. – Joe - GMapsBook.com Apr 08 '20 at 08:03
  • I had edited the file in Notepad++ , but thanks for the solution. – Murtaza Haji Apr 08 '20 at 21:12