Python Pycharm: Out of memory when loading large JSON files

Question

The following code executed in order to load several large (~200MB) JSON files:

def work():
    jsons = get_data()
    # do something with the jsons

def get_data():
    json_files = []
    for json_path in file_paths_list:
        json_files.append(load_json(json_path))
    return json_files

def load_json(json_path):
    import json
    with open(json_path) as f:
        return json.load(f)

This is how Pycharm's custom VM options look (up to 30GB heap size, RAM is 32GB):

# custom PyCharm VM options

-Xms25000m
-Xmx30000m
...
...
...

Popular recommendation to "Invalidate Caches/Restart" already applied.

After loading 2 files (total of ~400MB), during the 3rd, exception "MemoryError" thrown.

I cannot understand why if I have up to 30GB heap size, the memory error is thrown after only 400MB?

because the data loaded in memory takes more room than the text file itself, because of all metadata added to the objects. A string weighs more than its contents for instance. That said, 30Gb should be enough all right — Jean-François Fabre, Sep 14 '18 at 19:09
also have you tested with a command line python (64 bit). Pycharm doesn't execute anything. If your python version is 32 bt, that could be the issue. — Jean-François Fabre, Sep 14 '18 at 19:10
@Jean-FrançoisFabre But still - It exceeds 30GB? It's a lot. — michael, Sep 14 '18 at 19:10
check the version of your underlying python interpreter. I bet it's a 32 bit version. — Jean-François Fabre, Sep 14 '18 at 19:10

Jean-François Fabre · Accepted Answer · 2018-09-14T19:17:32.040

PyCharm is the Python IDE, not the Python interpreter. The memory it uses is for the editing phase.

400MB of files may well expand to several gigabytes of data (maybe not 30 but 3 or 4), because of the overhead of python objects. Example:

>>> s = "hello"
>>> import sys
>>> sys.getsizeof(s)
54

basically the size of the object in ram is much higher than the size of the string.

So if your python interpreter is a 32-bit interpreter, you have a 2GB or 3GB limit, which can explain this. PyCharm uses a 64-bit core but is unable to help with the interpreter part.

Upgrade to a 64 bit interpreter, which is able to benefit of all your RAM.

You can check the version info & 32/64 bit info with this (from Pycharm):

>>> import sys
>>> sys.version

For instance I get:

('3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit '
 '(AMD64)]')

If it shows "32 bit", my guess is correct. So uninstall the 32 bit version & just install the same version of python, but in 64 bit, and select it as the current interpreter in pycharm.

You may need to install additional modules in the new installation, so better dump the requirement textfile before uninstalling to be able to perform a global pip install on the new 64 bit version.

correct indeed I have 32, trying to figure out how to upgrade — michael, Sep 14 '18 at 19:17
just install the same version of python, but in 64 bit, and select it as the current interpreter in pycharm. For more safety, uninstall the 32 bit version. — Jean-François Fabre, Sep 14 '18 at 19:18
if you only have numpy to install, it's easy. The issue is when you don't remember all the packages you really need. — Jean-François Fabre, Sep 14 '18 at 21:39

Python Pycharm: Out of memory when loading large JSON files

1 Answers1

Linked