Pyarrow fs.HadoopFileSytem reports unable to load libhdfs.so

Question

I'm trying to use the pyarrow Filesystem interface with HDFS. I receive a libhdfs.so not found error when calling the fs.HadoopFileSystem constructor even though libhdfs.so is apparently at the indicated location.

from pyarrow import fs
hfs = fs.HadoopFileSystem(host="10.10.0.167", port=9870)

OSError: Unable to load libhdfs: /hadoop-3.3.1/lib/native/libhdfs.so: cannot open shared object file: No such file or directory

I've tried different python and pyarrow versions and setting ARROW_LIBHDFS_DIR. For testing, I'm using the following dockerfile on linuxmint.

FROM openjdk:11

RUN apt-get update &&\
  apt-get install wget -y

RUN wget -nv https://dlcdn.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1-aarch64.tar.gz &&\
  tar -xf hadoop-3.3.1-aarch64.tar.gz

ENV PATH=/miniconda/bin:${PATH}
RUN wget -nv https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh &&\
  bash miniconda.sh -b -p /miniconda &&\
  conda init 

RUN conda install -c conda-forge python=3.9.6
RUN conda install -c conda-forge pyarrow=4.0.1

ENV JAVA_HOME=/usr/local/openjdk-11
ENV HADOOP_HOME=/hadoop-3.3.1  

RUN  printf 'from pyarrow import fs\nhfs = fs.HadoopFileSystem(host="10.10.0.167", port=9870)\n' > test_arrow.py

# 'python test_arrow.py' fails with ... 
# OSError: Unable to load libhdfs: /hadoop-3.3.1/lib/native/libhdfs.so: cannot open shared object file: No such file or directory
RUN python test_arrow.py || true

CMD ["/bin/bash"]

score 0 · Answer 1 · answered Apr 18 '22 at 19:49

I have created a docker file for the pyarrow fs hadoopfilesystem client. HDFS needs to be installed to use the libhdfs.so file.

    RUN mkdir -p /data/hadoop
    RUN apt-get -q update
    RUN apt-get install software-properties-common -y
    RUN add-apt-repository "deb http://deb.debian.org/debian/ sid main"
    RUN apt-get -q update
    RUN apt-get install openjdk-8-jdk -y
    RUN apt-get clean
    RUN rm -rf /var/lib/apt/lists/*
    RUN wget "https://dlcdn.apache.org/hadoop/common/hadoop-3.3.2/hadoop-3.3.2.tar.gz" -O hadoop-3.3.2.tar.gz
    RUN tar xzf hadoop-3.3.2.tar.gz
    ENV HADOOP_HOME=/app/hadoop-3.3.2
    ENV HADOOP_INSTALL=$HADOOP_HOME
    ENV HADOOP_MAPRED_HOME=$HADOOP_HOME
    ENV HADOOP_COMMON_HOME=$HADOOP_HOME
    ENV HADOOP_HDFS_HOME=$HADOOP_HOME
    ENV YARN_HOME=$HADOOP_HOME
    ENV HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    ENV PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
    ENV HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/nativ"
    ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
    ENV CLASSPATH="$HADOOP_HOME/bin/hadoop classpath --glob"
    ENV ARROW_LIBHDFS_DIR=$HADOOP_HOME/lib/native
    ADD pyarrow-app.py /app/
    CMD [ "python3" "-u" "/app/pyarrow-app.py.py"]

Pyarrow fs.HadoopFileSytem reports unable to load libhdfs.so

1 Answers1