6

I'm on a Macbook with M1 (Apple ARM architecture) and I've tried running the following Python code using the layoutparser library, which indirectly uses pycocotools:

import layoutparser as lp
lp.Detectron2LayoutModel()

And I've received the error:

[...]
ImportError: 
dlopen([...]/.venv/lib/python3.9/site-packages/pycocotools/_mask.cpython-39-darwin.so, 0x0002): 
tried: 
'[...]/.venv/lib/python3.9/site-packages/pycocotools/_mask.cpython-39-darwin.so' 
(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), 
'/usr/local/lib/_mask.cpython-39-darwin.so' (no such file), 
'/usr/lib/_mask.cpython-39-darwin.so' (no such file)

The crucial info for me seems to be [...] is an incompatible architecture (have 'x86_64', need 'arm64e') [...]. Indeed, I am using the Apple ARM architecture, and sometimes it is not supported by some software. This is usually solved by using Rosetta, which simulates an Intel-x64 architecture. So I start a terminal with Rosetta (arch -x86_64 zsh), create a new virtual environment, make a fresh install of the dependencies, and try to run the code again ...

... and I receive the same error that I had also had without Rosetta:

[...] is an incompatible architecture (have 'x86_64', need 'arm64e') [...]

I've double-checked that Rosetta is really activated:

> uname -m
x86_64

Rosetta seems to be working. And yet, according to the error message, it seems not to be working.

Any ideas what could be the problem with Rosetta, or the library, or whatever, and how I could try fixing it?

David
  • 513
  • 7
  • 14
  • 3
    Do you actually have an x86_64 copy of Python? If all you have is an arm64 Python, it'll never be able to load an x86_64 shared library, no matter how your system identifies itself. – Charles Duffy Dec 06 '21 at 23:31
  • 3
    (Rosetta works at a process-by-process level; it doesn't allow executables of one architecture to load libraries of another: When your executable was built for one CPU, every library it loads needs to be compiled with support for that same CPU). – Charles Duffy Dec 06 '21 at 23:32
  • 3
    (output from `python -c 'import platform; print(platform.platform())'` would be good information to include in the question -- maybe also the banner from opening the REPL as well). – Charles Duffy Dec 06 '21 at 23:36
  • Thank you Charles, your comments are very helpful and fixed the problem! – David Dec 07 '21 at 00:05

1 Answers1

12

Charles Duffy explained the problem in the comments, thank you!

When I checked the platform in Python, it was indeed ARM:

> python -c 'import platform; print(platform.platform())'
macOS-12.0.1-arm64-i386-64bit

So I had been using a Python installation for ARM.

Now I installed brew and then python3 from the Rosetta terminal and used the newly installed Python to initiate a fresh virtual environment, and this fixed it. (This article helped me a bit with it.)

Update:

When creating Python environments with conda, it is possible to specify whether they should use Apple ARM or Intel-x64:

  • CONDA_SUBDIR=osx-arm64 conda create -n my_env python makes an ARM environment
  • CONDA_SUBDIR=osx-64 conda create -n my_env python makes an x64 environment
David
  • 513
  • 7
  • 14