2

I'm trying to use file-magic on Alpine Linux and it keeps blowing up with AttributeError: Symbol not found: magic_open whenever I import the magic module.

I noted that there's two Python modules out there with the same magic namespace, but as most Linux distros appear to be using file-magic and not python-magic, I decided to make my module dependent on the former. However on Alpine, only python-magic appears to work:

Setup Alpine & install libmagic:

$ docker run --rm -it python:3-alpine /bin/sh
/ # apk add libmagic
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/1) Installing libmagic (5.32-r0)
OK: 21 MiB in 35 packages

Install file-magic:

/ # pip install file-magic
Collecting file-magic
  Downloading https://files.pythonhosted.org/packages/bb/7e/b256e53a6558afd348387c3119dc4a1e3003d36030584a427168c8d72a7a/file_magic-0.4.0-py3-none-any.whl
Installing collected packages: file-magic
Successfully installed file-magic-0.4.0

It doesn't work though:

/ # python
Python 3.7.1 (default, Dec 21 2018, 03:21:42) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import magic
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/magic.py", line 61, in <module>
    _open = _libraries['magic'].magic_open
  File "/usr/local/lib/python3.7/ctypes/__init__.py", line 369, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/lib/python3.7/ctypes/__init__.py", line 374, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: Symbol not found: magic_open
>>>

Swap out file-magic for python-magic:

$ pip uninstall file-magic
Uninstalling file-magic-0.4.0:
  Would remove:
    /usr/local/lib/python3.7/site-packages/file_magic-0.4.0.dist-info/*
    /usr/local/lib/python3.7/site-packages/magic.py
Proceed (y/n)? y
  Successfully uninstalled file-magic-0.4.0

/ # pip install python-magic
Collecting python-magic
  Downloading https://files.pythonhosted.org/packages/42/a1/76d30c79992e3750dac6790ce16f056f870d368ba142f83f75f694d93001/python_magic-0.4.15-py2.py3-none-any.whl
Installing collected packages: python-magic
Successfully installed python-magic-0.4.15

It works:

/ # python
Python 3.7.1 (default, Dec 21 2018, 03:21:42) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import magic
>>>

I understand that every distro has it's quirks, and I want to accommodate, but I just don't know how to do that at the moment. Is there a way to define a module's setup.py to work around this situation? Is there a way to tweak the Docker container to work with file-magic? What's the right thing to do in this case?

Daniel Quinn
  • 6,010
  • 6
  • 38
  • 61

3 Answers3

4

Python-Magic has a hard-coded fallback for Alpine Linux so you might be in for a hard time with file-magic, which just uses ctypes.util.find_library('magic') and apparently can't find your library.

The exact behavior of ctypes.util.find_library is pretty complicated but if you trace through the code and see which methods it attempts to use on your platform you will probably find a way to get it to find the library (probably one of gcc, /sbin/ldconfig, or objdump).

Another solution is to patch ctypes.py to properly handle systems like Alpine:

wget https://patch-diff.githubusercontent.com/raw/python/cpython/pull/10461.patch -O - | patch /usr/local/lib/python3.7/ctypes/util.py

Now you just need to set LD_LIBRARY_PATH and Python can find the library:

/ # LD_LIBRARY_PATH="/usr/lib/" python -c "import ctypes.util; print(ctypes.util.find_library('magic'))"
libmagic.so.1
Blender
  • 289,723
  • 53
  • 439
  • 496
  • Wow. There must be an easier way than this, no? How do other packages that need libmagic manage to be portable across distros? – Daniel Quinn Dec 26 '18 at 21:05
  • 1
    @DanielQuinn: Alpine manages to have no combination of packages that allow `ctypes`'s `find_library` to work. [It's a known](https://github.com/docker-library/python/issues/111) [bug](https://bugs.alpinelinux.org/issues/4512) so you either can wait for a few months (or years in this case) for it to be fixed upstream or apply the patch yourself. – Blender Dec 26 '18 at 21:38
  • Wow. Well good to know! I guess I'll add your stuff above to my Dockerfile and include a reference to this in the documentation for my module. – Daniel Quinn Dec 26 '18 at 22:49
0

The other solution didn't work for me Docker Python 2.7 Alpine

Solution - rename the file :)

# fix no libmagic found error
RUN cp /usr/lib/libmagic.so.1 /usr/lib/libmagic.so
Eric Aya
  • 69,473
  • 35
  • 181
  • 253
Vic Barr
  • 1
0

apk update && apk add -y --no-install-recommends --no-cache libmagic worked for me.

You need to put this in the docker image.

PeterKogan
  • 21
  • 1