4

I have two JSON files (one about 40 mb and the second ~30 mb). I include them like this:

$one= json_decode(file_get_contents('../../trigrams.json'), true);
$two= json_decode(file_get_contents('../../fourgrams.json'), true);

On Default i get:

Allowed memory size of 134217728 bytes exhausted (tried to allocate 44 bytes)

I tried

ini_set('memory_limit', '1024M');

Which results in

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 3 bytes)

Could someone please explain these numbers? Why do I need so much memory?

EDIT: The marked duplicate does not solve the problem, since it doesn't explain why the memory usage is so hight for such a relativly small amount of data.

EDIT2: Error is gone when setting

ini_set('memory_limit', '2048M');

But this is not the wanted solution.

Frnak
  • 6,601
  • 5
  • 34
  • 67
  • thx but this doesn't explain why i need 1024M for 70MB of data - I'm pretty much aware of why I would need to increase the memory and that I fetch the data at once, but i actually need this for my purpose so I just want to get the clue behind why php needs such a large amount of memory for this task – Frnak Jan 25 '16 at 13:14
  • 3
    70MB is the size of the string representation of that data; json_decode converts that to a whole series of nested objects/arrays/scalars, each of which is an individual PHP zval with relevant overheads – Mark Baker Jan 25 '16 at 13:20
  • the php json decoder is very memory-consuming. I myself have ran into this and couldn't find the exact answer. Other languages are more optimized for this, but apparently, PHP has been left behind. @MarkBaker: +1 – Alex Tartan Jan 25 '16 at 13:20
  • Memory doesn't seem to be the problem, you raised it and still got the error? – Ruan Mendes Jan 25 '16 at 13:21
  • The error is gone when I raise it to > 1024M – Frnak Jan 25 '16 at 13:21
  • @JuanMendes - memory is still the problem, simply raising memory doesn't automatically solve memory issues unless you raise it be a large enough value – Mark Baker Jan 25 '16 at 13:22
  • check this: https://github.com/salsify/jsonstreamingparser – Federkun Jan 25 '16 at 13:33
  • The problem isn't with JSON parsing, but how php store it's so called hashmaps in memory. PHP tend to use so much memory to store simplest form of data, read https://nikic.github.io/2011/12/12/How-big-are-PHP-arrays-really-Hint-BIG.html and http://stackoverflow.com/questions/25081344/how-does-php-memory-actually-work so actually you can't do anything about it if you'r going to load lots of data in PHP other than increasing the allowed memory! – Ali Jan 25 '16 at 13:33
  • the JSON parsers that come with PHP will load the entire json in memory, so yeah, the problem is the JSON parsing – Federkun Jan 25 '16 at 13:38
  • @Federico If you are going to access different node on JSON frequently, parsing it from disk each time would be frustratingly slow. on the other hand loading 70 MB of JSON wouldn't impose such a memory overhead in other languages. So stream parsing is not a real solution and this is PHP problem that can't load lots of data. I suggest to use a mature database (Mongodb if you want JSON) if you'r going to use a stream parser! – Ali Jan 25 '16 at 13:43
  • yeah, I agree about that – Federkun Jan 25 '16 at 13:50

0 Answers0