0

as by title I have a beast of json files that I would like to split into smaller files. The size of this json file is 8.7 Gb. The format/json are located at this link: Detailed book graph . is the first paragraph. The file is big enough to saturate the RAM of my PC (32 GB) I tried to look for some tools online or on github, but nothing worked. Anyone have any idea how I can do that?

Zack
  • 15
  • 4
  • i havent looked at the exact json file contents, but often for big data, the file contains a json-record per line. This means that the complete file isnt strictly parseable as json, but the individual lines are (the file is not a big array of json objects). So if that is the case, you can try a file readstream in whichever programming language you like. – Flame May 11 '21 at 11:23
  • In my opinion, you shoud write a program for reading that file. I think you need to read line by line or with fixed buffer size. Also you can check out this question : https://stackoverflow.com/questions/2161895/reading-large-text-files-with-streams-in-c-sharp – ismailperim May 11 '21 at 11:26
  • Using pandas or csv I wrote a program that reads line by line the json file. Unfortunately after 5 minutes the compiler returns the error "Could not reserve memory block, " – Zack May 11 '21 at 13:44
  • The problem is that when I use functions like load json , python tries to load all the file information into RAM and then process it. At least I think he does that. – Zack May 11 '21 at 13:47

0 Answers0