Stack Overflow
Stack Overflow

Questions tagged [ijson]

Ijson is an iterative JSON parser with standard Python iterator interfaces.

Links

66 questions
6
votes
1 answer

ijson fails with trailing garbage parse error

for prefix, event, value in parser: print(prefix) I get the following error after executing the above code , I dont understand what the error is. ijson.common.IncompleteJSONError: parse error: trailing garbage …
Priya darshini M
  • 271
  • 3
  • 4
6
votes
4 answers

How to read a large JSON file using Python ijson?

I am trying to parse a big json file (hundreds of gigs) to extract information from its keys. For simplicity, consider the following example: import random, string # To create a random key def random_string(length): return…
Paul
  • 165
  • 1
  • 1
  • 13
4
votes
1 answer

Update JSON value with ijson in Python

I am working on a very big JSON file, and I decided to use the ijson library. Now I want to update some value in this JSON file, but I don't know how. For example, from: {"age": 25, "gender": "M"} update to: {"age": 30, "gender": "F"} by using the…
Luca Marchetti
  • 41
  • 2
3
votes
1 answer

Convert decimal to float with ijson.items

I am reading a big array of JSONS from a file with ijson.items , but numbers are converted to type Decimal. Example Decimal('14.2') The documentation says that there is an option use-float = true. But I don't know where to put this option. Is it…
Juan
  • 339
  • 3
  • 15
3
votes
3 answers

Parsing multiple json objects from a text file using Python

I have a .json file where each line is an object. For example, first two lines are: {"review_id":"x7mDIiDB3jEiPGPHOmDzyw","user_id":"msQe1u7Z_XuqjGoqhB0J5g","business_id":…
rohan
  • 527
  • 1
  • 6
  • 19
3
votes
2 answers

How to use ijson/other to parse this large JSON file?

I have this massive json file (8gb), and I run out of memory when trying to read it in to Python. How would I implement a similar procedure using ijson or some other library that is more efficient with large json files? import pandas as pd #There…
user9090553
  • 31
  • 1
  • 3
3
votes
0 answers

How to extract data from a 4GB JSON file?

I've got a 4GB JSON file with the following structure: { rows: [ { id: 1, names: { first: 'john', last: 'smith' }, dates: ...}, { id: 2, names: { first: 'tim', middle: ['james', 'andrew'], last: 'wilson' }, dates: ... }, …
Richard
  • 62,943
  • 126
  • 334
  • 542
2
votes
1 answer

Alternative Way to Load Large Json File

I am trying to load a large json file (around 4G) as a pandas dataframe, but the following method does not work for file > around 2G. Is there any alternative method? data_dir = 'data.json' my_data = pd.read_json(data_dir, lines = True) I tried…
Howell Yu
  • 73
  • 8
2
votes
2 answers

Is it possible to use ijson to create and load JSON objects in memory (not from/to a file)?

I'm trying to use ijson instead of json to be able to efficiently dump/load dictionaries to/from strings (in-memory, not from a file) [1]. Are there any examples for ijson analogous to standard dumping/loading with json? All sources I've seen that…
ballade4op52
  • 2,142
  • 5
  • 27
  • 42
2
votes
1 answer

load 1.4 GB json data into mysql using ijson python

I came across several thread talking about ijson to load huge JSON files in python, as this is the way to not consume all memory. My file is around 1.4 GB in size, it has several nodes (see below image), i am interested only in one node which hold…
Ammar Khwaira
  • 35
  • 1
  • 8
2
votes
0 answers

Parsing error in the middle of large (gigabytes) JSON file

I'm using ijson (https://pypi.python.org/pypi/ijson) to parse a large JSON file. It's several GBs, so I can't realistically store it all in memory. The issue is that somewhere in the middle of the file, the parser runs into an error (the specific…
nilypp
  • 21
  • 1
2
votes
3 answers

Load an element with python from large json file

So, here is my json file. I want to load the data list from it, one by one, and only it. And then, for exemple plot it... This is an exemple, because I am dealing with large data set, with wich I could not load all the file (that would create a…
Agape Gal'lo
  • 687
  • 4
  • 9
  • 23
2
votes
2 answers

Loading Large Twitter JSON Data (7GB+) into Python

I've set up a public stream via AWS to collect tweets and now want to do some preliminary analysis. All my data was stored on an S3 bucket (in 5mb files). I downloaded everything and merged all the files into one. Each tweet is stored as a standard…
shishy
  • 787
  • 1
  • 15
  • 31
2
votes
3 answers

Read top-level JSON dictionary incrementally using Python ijson

I have the following data in my JSON file: { "first": { "name": "James", "age": 30 }, "second": { "name": "Max", "age": 30 }, "third": { "name": "Norah", "age": 30 }, …
tuxdna
  • 8,257
  • 4
  • 43
  • 61
1
vote
0 answers

How could msgspec and ijson faster than json for large json file?

I was told that ijson create a pipe between the file on the hardrive and the memory to save the memory, so it's more memory efficient than json library. However, I was also told that ijson can be faster than json with the incremental parsing, which…
ShoutOutAndCalculate
  • 457
  • 5
  • 16
1
2 3 4 5