1

I was told that ijson create a pipe between the file on the hardrive and the memory to save the memory, so it's more memory efficient than json library. However, I was also told that ijson can be faster than json with the incremental parsing, which did not make sense to me at all. The hard drive's IO was usually much slower than the Ram's IO to out compete the cpu clock speed.

I have a 2 million entry json file around 3.5 GB to be loaded, and the data required very frequent search through the files. Can msgspec or ijson be faster than json? Or should I just load the json file into the RAM?

ShoutOutAndCalculate
  • 457
  • 5
  • 16
  • Sounds like something you could actually test directly in your use case rather than us guessing? – roganjosh May 14 '23 at 21:45
  • @roganjosh I haven't tried it out yet. I was trying to decide if I should extract the json data format to another class object or just use json directly. It's a lot of coding in different trends for the later functions so I wanted to ask for the experts' opinions to avoid some obvious mistakes. – ShoutOutAndCalculate May 14 '23 at 23:00

0 Answers0