how can i have a static build of lxml in Unix/Linux system?

Question

i have now two unix systems, one for service, one for build(having all environment to build, yet it's old). i need to use lxml in python in service machine. the following commands i tried:

python setup.py build --static-deps

or

CFLAGS="-g -O2 -fPIC"
python setup.py build --static-deps

but the result is:

ld: fatal: relocations remain against allocatable but non-writable sections
collect2: ld returned 1 exit status
error: command '/usr/lib/python2.6/pycc' failed with exit status 1

i am wondering how can i make a static build so that i could easily deploy to my service box?

additionally, if i

python setup.py build

it has no error, but if i:

Python 2.6.4 (r264:75706, Apr 17 2011, 11:24:50) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: ld.so.1: isapython2.6: fatal: relocation error: file /usr/lib/python2.6/site-packages/lxml/etree.so: symbol __xmlStructuredErrorContext: referenced symbol not found

i've searched it: get errors when import lxml.etree to python doesn't seem to have a good answer. i blame it's the linking problem, so that i think having it statically linked should be a better solution.

but my main goal is to reduce work spent in deployment so i will accept any easy ways.

please help. thanks in advance.

Answer 1

The problem appears to be that you're trying to build a static library without static dependencies.

A static library, like libxml2.a, is just an archive of objects (.o files). When you link against this, the linker copies the code for any functions you call out of those objects into your link target. So, the result will be a standalone target that doesn't require libxml2.a to run; when you call functions from libxml2, everything works because the code is in your library.

A shared library, like libxml2.so, is basically an executable. When you link against this, the linker creates relocation entries so that, when your target and libxml2.so are loaded into memory at the same time, any functions you call out of libxml2 will work, because the code is in libxml2.so.

What you want to do here is actually to build a hybrid: a shared library (so it can be loaded as a module) that probably dynamically links to libpython.so (and maybe other things that are built in to the OS), but that statically links in the code from libxml2 (and libxslt2).

A modern linker has absolutely no problem doing that—but it needs the static libs for libxml2 to do that.

So, you need to get libxml2.a (and libxslt2.a, and maybe even libz.a) installed before you can build what you want.

---

I've oversimplified a little bit here. On some platforms, a .so file is actually just a front-end to the real shared object, basically the relocation tables plus a link to the real file. On other platforms, code can be pulled out of .so files and "relocated" in-place, and .a files can just be similar front-ends to shared objects. And there are platforms that are so different from ELF that this terminology is all more misleading than useful.

If you want to learn more, start with this question and other linked and related questions from there.

If you don't, just take it on faith that you need the .a files for anything you want to statically link.

---

If you read the installation docs, you'll notice that:

On Linux (and most other well-behaved operating systems), pip will manage to build the source distribution as long as libxml2 and libxslt are properly installed, including development packages, i.e. header files, etc. Use your package management tool to look for packages like libxml2-dev or libxslt-devel if the build fails, and make sure they are installed. Alternatively, setting STATIC_DEPS=true will download and build both libraries automatically.

So, if you don't have those libraries at all, building lxml will download the dependencies, build the .a files for you, and link against them.

But if you do have them, then (a) pkg-config and/or xml2-config is going to find them and use them, and (b) even if it doesn't, lxml may end up building with -lxml2 instead of an absolute path to libxml2.a and link the wrong one anyway.

How can you get around that? Well, you'd think there would be a simple flag that would tell it not to do that, but as far as I can tell, there isn't. So, your options include:

• Build with a fake root that doesn't contain libxml2 and libxslt2. This is loosely hinted at in the Building Debian packages docs.
• Manually build static-only libxml2 and libxslt2 and pass the absolute paths to the resulting xml2-config and xslt2-config to the setup.py, as explained in Building the sources.
• Temporarily hack up xml2-config and xslt2-config (or, if you don't have those, the .pc files used by pkg-config) to use an absolute path to the .a files in LDFLAGS, instead of the -L and -l flags.
• Temporarily get the .so files off your library path entirely while building lxml (by renaming them, by setting a different path, or otherwise).
• Assuming the version numbers are (or can be) different from those of the .so files you have installed, you can cheat by following the Mac build instructions to specify explicit version numbers. You may have to download the tarballs manually for this to work, but you won't have to build them.

If this all seems like kind of a pain, and like something that should be easily… well, remember what you're trying to do here. Linux is designed around the idea of building source distributions the best way possible for your particular system. If your system has the shared libs for xml2 and xslt2, everything is set up to try to use them. If you want to build code to deploy to a system that doesn't have those, your development system is expected not to have them either; if it does, you're expected to know how to work around that.