MemoryError while reading JSON from txt file

Question

I have 3 txt files (file1.txt, file2.txt, file3.txt), that contains JSON formatted data. The size of the files like something like this:

file1 = 104.55 Mb
file2 = 104.68 Mb
file3 = 104.49 Mb

I can open and read file1 and file3 without any problems, but when I try to open file2 I'm getting a MemoryError despite its size is almost the same as the others, the difference is less than 200KB.

Here is my implementation for opening the files:

with open(file1.txt, encoding="utf-8") as f:
    jsonContent = json.load(f)

json_data = json.dumps(jsonContent)
file_data = json.loads(json_data)

I would really appreciate if somebody could tell me what can cause this issue, because everything seems fine for me and I can't believe that 200KB is the reason. Am I doing something wrong in my code or it is what it looks like, I don't have enough memory?

Do you want to load all of them together? – Mazdak Apr 10 '16 at 13:58 — Mazdak, Apr 10 '16 at 13:58
@Kasramvd No. I would like to load one file always. – gatto Apr 10 '16 at 14:10 — gatto, Apr 10 '16 at 14:10

score 0 · Answer 1 · edited May 23 '17 at 11:45

0

It's always hard to say exactly what's causing the issue; but this question has been asked before, usually replied to with links to iJSON or json streamer.

I won't say that's the defacto solution for you, but the question is common. If you go the iJSON route, please look at this post and the thread around it.

edited May 23 '17 at 11:45

Community

1
1

answered Apr 10 '16 at 14:45

Wolfie

143
6

MemoryError while reading JSON from txt file

1 Answers1