3

I am having an issue importing pyarrow into my jupyter notebook. To give some context, I created a virtual environment named rc_env. I've uninstalled and reinstalled pyarrow quite a few times but this the message I received when I recently install it.

pip install pyarrow
Requirement already satisfied: pyarrow in ./Environments/rc_env/lib/python3.8/site-packages (4.0.1)
Requirement already satisfied: numpy>=1.16.6 in ./Environments/rc_env/lib/python3.8/site-packages (from pyarrow) (1.21.0)

So the package is in my environment. And for another sanity check, I used the pip list command to see the python packages installed in my environment.

(rc_env) LTA00015JFK:~ adenner$ pip list
Package                       Version
----------------------------- ---------
appnope                       0.1.2
argon2-cffi                   20.1.0
async-generator               1.10
attrs                         21.2.0
backcall                      0.2.0
beautifulsoup4                4.9.3
bleach                        3.3.0
boto3                         1.17.103
botocore                      1.20.103
cachetools                    4.2.2
certifi                       2021.5.30
cffi                          1.14.5
chardet                       4.0.0
cycler                        0.10.0
debugpy                       1.3.0
decorator                     5.0.9
defusedxml                    0.7.1
dictor                        0.1.7
entrypoints                   0.3
et-xmlfile                    1.1.0
google                        3.0.0
google-api-core               1.30.0
google-api-python-client      2.11.0
google-auth                   1.32.1
google-auth-httplib2          0.1.0
google-auth-oauthlib          0.4.4
google-cloud                  0.34.0
google-cloud-bigquery         2.20.0
google-cloud-bigquery-storage 2.5.0
google-cloud-core             1.7.1
google-cloud-vision           2.3.2
google-crc32c                 1.1.2
google-resumable-media        1.3.1
googleapis-common-protos      1.53.0
grpcio                        1.38.1
httplib2                      0.19.1
idna                          2.10
ipykernel                     6.0.0
ipython                       7.25.0
ipython-genutils              0.2.0
ipywidgets                    7.6.3
jedi                          0.18.0
Jinja2                        3.0.1
jmespath                      0.10.0
jsonschema                    3.2.0
jupyter                       1.0.0
jupyter-client                6.1.12
jupyter-console               6.4.0
jupyter-core                  4.7.1
jupyterlab-pygments           0.1.2
jupyterlab-widgets            1.0.0
kiwisolver                    1.3.1
libcst                        0.3.19
MarkupSafe                    2.0.1
matplotlib                    3.4.2
matplotlib-inline             0.1.2
mistune                       0.8.4
mypy-extensions               0.4.3
nbclient                      0.5.3
nbconvert                     6.1.0
nbformat                      5.1.3
nest-asyncio                  1.5.1
notebook                      6.4.0
numpy                         1.21.0
oauthlib                      3.1.1
openpyxl                      3.0.7
packaging                     20.9
pandas                        1.2.5
pandas-gbq                    0.15.0
pandocfilters                 1.4.3
parso                         0.8.2
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        8.3.0
pip                           21.1.3
prometheus-client             0.11.0
prompt-toolkit                3.0.19
proto-plus                    1.19.0
protobuf                      3.17.3
ptyprocess                    0.7.0
pyarrow                       4.0.1
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.20
pydata-google-auth            1.2.0
Pygments                      2.9.0
pyparsing                     2.4.7
pyrsistent                    0.18.0
python-dateutil               2.8.1
pytz                          2021.1
PyYAML                        5.4.1
pyzmq                         22.1.0
qtconsole                     5.1.1
QtPy                          1.9.0
requests                      2.25.1
requests-oauthlib             1.3.0
rsa                           4.7.2
s3transfer                    0.4.2
scipy                         1.7.0
seaborn                       0.11.1
Send2Trash                    1.7.1
seshat                        0.8.5
setuptools                    57.0.0
six                           1.16.0
soupsieve                     2.2.1
terminado                     0.10.1
testpath                      0.5.0
titlecase                     2.2.0
tornado                       6.1
tqdm                          4.61.1
traitlets                     5.0.5
typing-extensions             3.10.0.0
typing-inspect                0.7.1
uritemplate                   3.0.1
urllib3                       1.26.6
uuid                          1.30
wcwidth                       0.2.5
webencodings                  0.5.1
wheel                         0.36.2
widgetsnbextension            3.5.1
XlsxWriter                    1.4.3

Now when I open up python and try to import the module, I receive the following error message:

(rc_env) LTA00015JFK:~ adenner$ python 
Python 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/adenner/Environments/rc_env/lib/python3.8/site-packages/pyarrow/__init__.py", line 63, in <module>
    import pyarrow.lib as _lib
ImportError: dlopen(/Users/adenner/Environments/rc_env/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /Users/adenner/Environments/rc_env/lib/python3.8/site-packages/pyarrow/libarrow.400.dylib
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/adenner/Environments/rc_env/lib/python3.8/site-packages/pyarrow/libarrow.400.dylib

I have tried using both anaconda3 and miniconda3 environments but unfortunately, I keep receiving the same error message. I am also using macOS High Serra version 10.13.6 Any suggestions?

Adrian Denner
  • 31
  • 1
  • 2
  • I just did `pip install pyarrow` in a new environment (created as `conda create -n pyarrow python=3.8`) and I did not have issues running `python -c "import pyarrow"`. Next, I installed jupyter lab (using conda, `conda install jupyterlab`) and I could do the same in the notebook environment. Can you try creating a new environment and see if that helps? – TDrabas Jul 01 '21 at 20:45
  • Since you're on Mac you might also want to try this: https://stackoverflow.com/questions/35029029/jupyter-notebook-command-does-not-work-on-mac – TDrabas Jul 01 '21 at 21:01

3 Answers3

3

This is a known bug with the latest pyarrow release: https://issues.apache.org/jira/browse/ARROW-13108. you will need macOS 10.15+ to run the wheel builds.

As you are already in an environment created by conda, you could instead use the pyarrow conda package. This will work on macOS 10.9+ and is even the preferred way to install pyarrow:

conda install -c conda-forge pyarrow

Alternatively using mamba:

mamba install -c conda-forge pyarrow
Uwe L. Korn
  • 8,080
  • 1
  • 30
  • 42
0

you can try the jupyter lab instead of using jupyter notebook. I got this problem when i use notebook, but pyarrow works well in jupyter lab

Ray
  • 73
  • 4
  • 1
    Pleae make more obvious what additional insight this post provides in comparison to an existing answer, e.g. https://stackoverflow.com/a/68225189/7733418 Also, you might enjoy using this info https://stackoverflow.com/editing-help – Yunnosch Oct 19 '21 at 05:51
0

In my case I was having problems with import pyarrow.parquet as pq when running my code on jupyter-lab. When I tried the same code as script as in if __name__ == __main__: my_func() , where my_function() had the import, then it worked. When I re-installed jupyter-lab the problems disappeared also in jupyter. conda install -c conda-forge jupyterlab

Gilberto
  • 1
  • 2