Pip install killed - out of memory - how to get around it?

Question

I'm trying to install spaCy, a library for natural language processing.

root@vps:~# python3 -m pip install spacy
Collecting spacy
  Using cached https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd29384247babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: spacy
Killed

After a while, it's killed. I've already looked for answers on Stack Overflow and tried, and it's due to low memory (1,2 Gb free on my VPS, can't extend). A quick tail -f /var/log/kern.log gives me Out of memory: Kill process 31729 (python3) score 558 or sacrifice child.

I've already tried adding --no-cache-dir, but it didn't help. Something seems missing when copying the files manually. Is there any way to install this package on a low-RAM server?

EDIT : Complete log for python3 -m pip install spacy -vvv

Converted retries value: Retry(total=5, connect=None, read=None, redirect=None)                                                                                                              -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=N                                                                                                             one, read=None, redirect=None)
Converted retries value: Retry(total=5, connect=None, read=None, redirect=None)                                                                                                              -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=N                                                                                                             one, read=None, redirect=None)
Collecting spacy
  1 location(s) to search for versions of spacy:
  * https://pypi.python.org/simple/spacy/
  Getting page https://pypi.python.org/simple/spacy/
  Looking up "https://pypi.python.org/simple/spacy/" in the cache
  Returning cached "301 Moved Permanently" response (ignoring date and etag info                                                                                                             rmation)
  Looking up "https://pypi.org/simple/spacy/" in the cache
  Current age based on date: 29458
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  Starting new HTTPS connection (1): pypi.org
  https://pypi.org:443 "GET /simple/spacy/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/spacy/
    <!-- Found link, skipped linked (hundreds of lines) -->
    Found link https://files.pythonhosted.org/packages/a7/84/f35f44560f0ab9ef23c                                                                                                             4a334a513f2b6c8a95d7b3347ef68a0247da63c08/spacy-2.1.6.tar.gz#sha256=b708bb582597                                                                                                             6220de78ad7e20e827912ca95653738a812cc2e5d194e3060ae8 (from https://pypi.org/simp                                                                                                             le/spacy/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 2.1                                                                                                             .6
  Using version 2.1.6 (newest of versions: 0.31, 0.32, 0.33, 0.40, 0.51, 0.52, 0                                                                                                             .60, 0.61, 0.62, 0.63, 0.64, 0.65, 0.67, 0.68, 0.70, 0.80, 0.81, 0.82, 0.83, 0.8                                                                                                             4, 0.85, 0.86, 0.87, 0.88, 0.89, 0.90, 0.91, 0.92, 0.93, 0.94, 0.95, 0.97, 0.98,                                                                                                              0.99, 0.100.0, 0.100.1, 0.100.2, 0.100.3, 0.100.4, 0.100.5, 0.100.6, 0.100.7, 0                                                                                                             .101.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.2.0, 1.3.0, 1.                                                                                                             4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.5, 1.8.0, 1.8.1, 1.8.2                                                                                                             , 1.9.0, 1.10.0, 1.10.1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7,                                                                                                              2.0.8, 2.0.9, 2.0.10, 2.0.11, 2.0.12, 2.0.13, 2.0.15, 2.0.16, 2.0.17, 2.0.18, 2                                                                                                             .1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6)
  Looking up "https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd29384247                                                                                                             babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_6                                                                                                             4.whl" in the cache
  Current age based on date: 98439
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 98439
  Using cached https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd2938424                                                                                                             7babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_                                                                                                             64.whl
  Downloading from URL https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3b                                                                                                             d29384247babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylin                                                                                                             ux1_x86_64.whl#sha256=2441ad9e6195669c13f4035dd77c4206414a40ebb8543c69f3386d984a                                                                                                             bfb942 (from https://pypi.org/simple/spacy/) (requires-python:>=2.7,!=3.0.*,!=3.                                                                                                             1.*,!=3.2.*,!=3.3.*)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.5/di                                                                                                             st-packages (from spacy)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.5/d                                                                                                             ist-packages (from spacy)
Requirement already satisfied: srsly<1.1.0,>=0.0.6 in /usr/local/lib/python3.5/d                                                                                                             ist-packages (from spacy)
Requirement already satisfied: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.5/di                                                                                                             st-packages (from spacy)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3                                                                                                             .5/dist-packages (from spacy)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/pytho                                                                                                             n3.5/dist-packages (from spacy)
Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.5/                                                                                                             dist-packages (from spacy)
Requirement already satisfied: thinc<7.1.0,>=7.0.8 in /usr/local/lib/python3.5/d                                                                                                             ist-packages (from spacy)
Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.5                                                                                                             /dist-packages (from spacy)
Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.5/dist-pa                                                                                                             ckages (from spacy)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/l                                                                                                             ocal/lib/python3.5/dist-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.5/dist-p                                                                                                             ackages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.5/di                                                                                                             st-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.5                                                                                                             /dist-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /usr/local/lib/python3.5/d                                                                                                             ist-packages (from thinc<7.1.0,>=7.0.8->spacy)
Installing collected packages: spacy
Killed

This is odd since `spacy` is available as wheel for your platform, so the installation is a simple unpacking of a zip archive. May be one of the dependencies, though. Can you run `python3 -m pip install spacy -vvv` and add the complete log to the question? — hoefling, Jul 16 '19 at 20:24
Have you tried adding swap space to your VPS? That might help depending on how much disk space you have and how much more memory you need — prithajnath, Jul 19 '19 at 21:16
@prithajnath It works! I added swap space, made it, thanks a lot :) Seems like I can't award a bounty on a comment. May you post an answer? — Rasgel, Jul 19 '19 at 21:33

score 23 · Accepted Answer · answered Jul 20 '19 at 01:28

This has been reported already on github

https://github.com/explosion/spaCy/issues/3396

Below are some good thread on how to solve the issue

fork() failing with Out of memory error

Memory error while using pip install Matplotlib

out of memory issue in installing packages on Ubuntu server

https://chirale.org/2017/01/15/memory-error-on-pip-install-solved/

In Summary, below are the two options:

Try with --no-cache-dir, which unfortunately has not helped you
Increase the swap space

$ create swap file of 512 MB
dd if=/dev/zero of=/swapfile bs=1024 count=524288

$ modify permissions
chown root:root /swapfile
chmod 0600 /swapfile

$ setup swap area
mkswap /swapfile

$ turn swap on
swapon /swapfile

score 3 · Answer 2 · answered Jul 19 '19 at 23:22

3

Try adding swap space to your VPS. That will let you use free disk space as memory

answered Jul 19 '19 at 23:22

prithajnath

2,000
14
17

I have to wait 9 hours before awarding the bounty ;) – Rasgel Jul 20 '19 at 11:09

score 0 · Answer 3 · answered Jul 19 '19 at 21:22

not sure if this will work but have you tried compiling from source? this script is at Spacy's pypi page. No idea if this would manage RAM constraint better than regular pip install but it's something I would try.

# make sure you are using the latest pip
python -m pip install -U pip
git clone https://github.com/explosion/spaCy
cd spaCy

python -m venv .env
source .env/bin/activate
export PYTHONPATH=`pwd`
pip install -r requirements.txt
python setup.py build_ext --inplace

Thanks a lot for your answer. Unfortunately, I had already tried it, and it did not work either. Swap space made it! — Rasgel, Jul 19 '19 at 21:31

Pip install killed - out of memory - how to get around it?

3 Answers3

Linked