Install lxml on OpenShift with python packages only (pip, easy_install)

Question

I am using Beautifulsoup with lxml package in my Django app (python 2.7). My production server is OpenShift, so I can not install using apt-get due to access rights. I was able to install lxml to OpenShift in past using easy_install in past, but I struggle to do so now.
Well probably I am missing some dependences that are installed through apt-get like these?
I tried to install lxml by adding lxml to my setup.py file. Did not work.
So I tried pip install lxml==3:

> pip install lxml==3
Downloading/unpacking lxml==3
  Downloading lxml-3.0.tar.gz (3.2MB): 3.2MB downloaded
  Running setup.py egg_info for package lxml
    Building lxml version 3.0.
    Building without Cython.
    Using build configuration of libxslt 1.1.26
    Building against libxml2/libxslt in the following directory: /usr/lib64

    warning: no previously-included files found matching '*.py'
    warning: no files found matching '*.txt' under directory 'src/lxml/tests'
Installing collected packages: lxml
  Running setup.py install for lxml
    Building lxml version 3.0.
    Building without Cython.
    Using build configuration of libxslt 1.1.26
    Building against libxml2/libxslt in the following directory: /usr/lib64
    building 'lxml.etree' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -I/var/lib/openshift/569eccb30c1e661fc20000d5/app-root/runtime/dependencies/python/virtenv/build/lxml/src/lxml/includes -I/opt/rh/python27/root/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o
    src/lxml/lxml.etree.c: In function ‘__pyx_pf_4lxml_5etree_4XSLT_18__call__’:
    src/lxml/lxml.etree.c:132418: warning: passing argument 1 of ‘__pyx_f_4lxml_5etree_12_XSLTContext__copy’ from incompatible pointer type
    src/lxml/lxml.etree.c:130379: note: expected ‘struct __pyx_obj_4lxml_5etree__XSLTContext *’ but argument is of type ‘struct __pyx_obj_4lxml_5etree__BaseContext *’
    src/lxml/lxml.etree.c: In function ‘__pyx_f_4lxml_5etree__copyXSLT’:
    src/lxml/lxml.etree.c:133807: warning: passing argument 1 of ‘__pyx_f_4lxml_5etree_12_XSLTContext__copy’ from incompatible pointer type
    src/lxml/lxml.etree.c:130379: note: expected ‘struct __pyx_obj_4lxml_5etree__XSLTContext *’ but argument is of type ‘struct __pyx_obj_4lxml_5etree__BaseContext *’
    src/lxml/lxml.etree.c: At top level:
    src/lxml/lxml.etree.c:11938: warning: ‘__pyx_f_4lxml_5etree_displayNode’ defined but not used
Connection to app-nick.rhcloud.com closed by remote host.
Connection to app-nick.rhcloud.com closed.

if I specify subversion (3.4 instead of 3) with pip there is a little bit different output:

pip install lxml==3.4
Downloading/unpacking lxml==3.4
  Downloading lxml-3.4.0.tar.gz (3.5MB): 3.5MB downloaded
  Running setup.py egg_info for package lxml
    /opt/rh/python27/root/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    Building lxml version 3.4.0.
    Building without Cython.
    Using build configuration of libxslt 1.1.26
    Building against libxml2/libxslt in the following directory: /usr/lib64

    warning: no previously-included files found matching '*.py'
Installing collected packages: lxml
  Running setup.py install for lxml
    /opt/rh/python27/root/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    Building lxml version 3.4.0.
    Building without Cython.
    Using build configuration of libxslt 1.1.26
    Building against libxml2/libxslt in the following directory: /usr/lib64
    building 'lxml.etree' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -I/var/lib/openshift/569eccb30c1e661fc20000d5/app-root/runtime/dependencies/python/virtenv/build/lxml/src/lxml/includes -I/opt/rh/python27/root/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w

and easy_install:

> easy_install lxml
Searching for lxml
Reading https://pypi.python.org/simple/lxml/
Best match: lxml 3.5.0
Downloading https://pypi.python.org/packages/source/l/lxml/lxml-3.5.0.tar.gz#md5=9f0c5f1eb43ff44d5455dab4b4efbe73
Processing lxml-3.5.0.tar.gz
Writing /tmp/easy_install-K7xT2d/lxml-3.5.0/setup.cfg
Running lxml-3.5.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-K7xT2d/lxml-3.5.0/egg-dist-tmp-BAe44x
Building lxml version 3.5.0.
Building without Cython.
Using build configuration of libxslt 1.1.26
Building against libxml2/libxslt in the following directory: /usr/lib64
Connection to app-nick.rhcloud.com closed by remote host.
Connection to app-nick.rhcloud.com closed.

How do I install lxml through pip or easy_install only? Or should I use another xml parser with BeautifulSoup (not sure if there are alternatives)?

UPDATE1

I tried to install lxml==3.2.5. I forgot to mention I am on python 2.7 gear. If I add , 'lxml==3.2.5' to my setup.py file(I think setup.py works the same as requirements.txt), this is output:

remote: Searching for lxml==3.2.5
remote: Reading http://mirror1.ops.rhcloud.com/mirror/python/web/simple/lxml/
remote: Best match: lxml 3.2.5
remote: Downloading http://mirror1.ops.rhcloud.com/mirror/python/web/packages/source/l/lxml/lxml-3.2.5.tar.gz#md5=6c4fb9b1840631cff09b8229a12a9ef7
remote: Processing lxml-3.2.5.tar.gz
remote: Writing /tmp/easy_install-jvJHz8/lxml-3.2.5/setup.cfg
remote: Running lxml-3.2.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-jvJHz8/lxml-3.2.5/egg-dist-tmp-KMpa_f
remote: /opt/rh/python27/root/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
remote:   warnings.warn(msg)
remote: warning: no files found matching '*.txt' under directory 'src/lxml/tests'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_4data':
remote: src/lxml/lxml.etree.c:97814: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxData' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97415: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_6start':
remote: src/lxml/lxml.etree.c:97951: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxStart' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97115: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_8end':
remote: src/lxml/lxml.etree.c:98009: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxEnd' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97339: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_10pi':
remote: src/lxml/lxml.etree.c:98167: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxPi' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97455: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_11TreeBuilder_12comment':
remote: src/lxml/lxml.etree.c:98220: warning: passing argument 1 of '__pyx_f_4lxml_5etree_11TreeBuilder__handleSaxComment' from incompatible pointer type
remote: src/lxml/lxml.etree.c:97581: note: expected 'struct __pyx_obj_4lxml_5etree_TreeBuilder *' but argument is of type 'struct __pyx_obj_4lxml_5etree__SaxParserTarget *'
remote: src/lxml/lxml.etree.c: In function '__pyx_f_4lxml_5etree__xslt_resolve_from_python':
remote: src/lxml/lxml.etree.c:136117: warning: enumeration value '__pyx_e_4lxml_5etree_PARSER_DATA_INVALID' not handled in switch
remote: src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_4XSLT_18__call__':
remote: src/lxml/lxml.etree.c:140191: warning: passing argument 1 of '__pyx_f_4lxml_5etree_12_XSLTContext__copy' from incompatible pointer type
remote: src/lxml/lxml.etree.c:138126: note: expected 'struct __pyx_obj_4lxml_5etree__XSLTContext *' but argument is of type 'struct __pyx_obj_4lxml_5etree__BaseContext *'
remote: src/lxml/lxml.etree.c: In function '__pyx_f_4lxml_5etree__copyXSLT':
remote: src/lxml/lxml.etree.c:141589: warning: passing argument 1 of '__pyx_f_4lxml_5etree_12_XSLTContext__copy' from incompatible pointer type
remote: src/lxml/lxml.etree.c:138126: note: expected 'struct __pyx_obj_4lxml_5etree__XSLTContext *' but argument is of type 'struct __pyx_obj_4lxml_5etree__BaseContext *'
Connection to app-host.rhcloud.com closed by remote host.
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer

UPDATE2 I tried to install Cython (pip install cython), it also failed -> I was disconnected from OpenShift.
I have created bug report in RedHat's bugzilla.

Did you try adding `lxml` in `requirements.txt` and pushing? That seems to install lxml 3.4.4 just fine for me with the Python 3.3 cartridge. — Jiri Fiala, Jan 21 '16 at 11:39
I install lxml setting the "requires" field in setup.py file "lxml==3.2.5" for my python 3.3 cartridge. There was a bug in Openshift a about a year or so ago that would only work with version 3.2.5. From what the previous poster states it looks like they might have fixed it. — fat fantasma, Jan 21 '16 at 12:25
hm, I am running on python 2.7. I will try to install your versions. Thanks — Lucas03, Jan 21 '16 at 16:03
@fatfantasma I added output from trying to install lxml version 3.2.5. — Lucas03, Jan 21 '16 at 16:29
Maybe it's a python 2.7 issue. I use lxml and beautifulsoup in all of my Openshift apps with python 3.3 — fat fantasma, Jan 21 '16 at 18:36
Yep, but that is weird. I do run lxml on my other python 2.7 app. But I wanted to move that app to scalable, so I needed to create new. And I can find out how I was able to install lxml before. — Lucas03, Jan 21 '16 at 19:36
Does it work for you with a python3.3 app from scratch? I [can't install lxml](http://stackoverflow.com/questions/35529761/openshift-cant-install-lxml-for-python-app). I see that you created a bug report on RedHat's bugzilla. I will ask there too to get someone from openshift in the loop. — Michael, Feb 21 '16 at 01:25

score 0 · Accepted Answer · answered Jan 26 '16 at 23:00

0

Weird bug, Maciej from RedHat suggested I recreate Openshift app again from scratch. It worked and I was able to install lxml successfully.

answered Jan 26 '16 at 23:00

Lucas03

2,267
2
32
60

score 0 · Answer 2 · answered Mar 29 '16 at 19:23

For the UPDATE1 symptoms, except not giving the version number in the requirements.txt as it was a dependency for another library I was using, this fixed it for me:

rhc ssh to the application
run pip install lxml
run pip install lxml again

After Step 2. (the first pip run) the output was:

Downloading/unpacking lxml

pip can't proceed with requirement 'lxml' due to a pre-existing build directory.
 location: /var/lib/openshift/xxx/app-root/runtime/dependencies/python/virtenv/build/lxml
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
Please delete it and try again.

Cleaning up...

The second pip run finished successfully and there was no need to delete the directory pip suggested (nor did it exist).

pip was using lxml version 3.6.0 - python 2.7 as cartridge.

Install lxml on OpenShift with python packages only (pip, easy_install)

2 Answers2