How to install PDAL library in databrisk cluster?

Question

I have installed the PDAL library ( https://pdal.io/en/latest/ ) on databricks cluster by using a databricks cluster based on container image. The issue is that the pdal library is not working properly, i am not able to translate to some of the data formats. For example, for las, i am not able to convert from obj to las, but conversion from obj to pcd or text is working.

I have installed pdal from conda in container using runtime9.x.

This is the docker file:

FROM databricksruntime/standard:9.x

# Install miniconda
ENV CONDA_DIR /databricks/conda

RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py38_23.1.0-1-Linux-x86_64.sh -O ~/miniconda.sh && \
     /bin/bash ~/miniconda.sh -b -p /databricks/conda 

COPY env.yml /databricks/.conda-env-def/env.yml

RUN /databricks/conda/bin/conda env create --file /databricks/.conda-env-def/env.yml \
    && ln -s /databricks/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
RUN /databricks/conda/bin/conda config --system --set channel_priority strict \
    && /databricks/conda/bin/conda config --system --set always_yes True
    
ENV DEFAULT_DATABRICKS_ROOT_CONDA_ENV=dcs-minimal
ENV DATABRICKS_ROOT_CONDA_ENV=dcs-minimal

the env.yml file:
name: dcs-minimal
channels:
  - conda-forge
dependencies:
  - python=3.8.8
  - pip=20.2.4
  - six=1.15.0
  - ipython=7.19.0
  - nomkl=3.0
  - numpy=1.19.2
  - pandas=1.1.5
  - traitlets=5.0.5
  - wheel=0.35.1
  - pdal 
  - python-pdal 
  - pip:
    - pyarrow==1.0.1

the conda list from the databricks cluster:

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
blas                      2.117                  openblas    conda-forge
blas-devel                3.9.0           17_linux64_openblas    conda-forge
blosc                     1.21.4               h0f2a231_0    conda-forge
boost-cpp                 1.78.0               h5adbc97_2    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cairo                     1.16.0               hb05425b_5  
cfitsio                   4.2.0                hd9d235c_0    conda-forge
curl                      7.87.0               h6312ad2_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
draco                     1.5.6                hf52228f_0    conda-forge
eigen                     3.4.0                h00ab1b0_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
freexl                    1.0.6                h166bdaf_1    conda-forge
geos                      3.11.1               h27087fc_0    conda-forge
geotiff                   1.7.1                h7a142b4_6    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
glib                      2.69.1               h4ff587b_1  
hdf4                      4.2.15               h9772cbc_5    conda-forge
hdf5                      1.12.2          nompi_h2386368_101    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
ipython                   7.19.0           py38h81c977d_2    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.17.2           py38h578d9bd_2    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
json-c                    0.16                 hc379101_0    conda-forge
kealib                    1.5.0                ha7026e8_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               hf9c8cef_0    conda-forge
lcms2                     2.15                 hfd0df8a_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libcurl                   7.87.0               h6312ad2_0    conda-forge
libdeflate                1.17                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgdal                   3.6.2                hae1867e_4    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libkml                    1.3.0             h37653c0_1015    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
liblapacke                3.9.0           17_linux64_openblas    conda-forge
libnetcdf                 4.8.1           nompi_h261ec11_106    conda-forge
libnghttp2                1.51.0               hdcd2b5c_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.1                 h2baec63_3    conda-forge
librttopo                 1.1.0               ha49c73b_12    conda-forge
libspatialite             5.0.1               h221c8f1_23    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libtiff                   4.5.0                h6adf6a1_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.1                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxml2                   2.10.3               hca2bb57_4    conda-forge
libzip                    1.9.2                hc869a4a_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
llvm-openmp               16.0.6               h4dfa4b3_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nitro                     2.7.dev8             h59595ed_0    conda-forge
nomkl                     3.0                           0  
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
numpy                     1.19.2           py38hf89b668_1    conda-forge
openblas                  0.3.23          pthreads_h855a84d_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   1.1.1u               hd590300_0    conda-forge
pandas                    1.1.5            py38h51da96c_0    conda-forge
parso                     0.7.1              pyh9f0ad1d_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pdal                      2.4.3                hdfc1b4a_5    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       20.2.4                     py_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
poppler                   22.12.0              h381b16e_0  
poppler-data              0.4.12               hd8ed1ab_0    conda-forge
postgresql                15.1                 h84e8d4a_3    conda-forge
proj                      9.1.1                h8ffa02c_2    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pyarrow                   1.0.1                    pypi_0    pypi
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
python                    3.8.8           hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-pdal               3.1.2            py38h514daf8_0    conda-forge
python_abi                3.8                      3_cp38    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
setuptools                59.8.0           py38h578d9bd_1    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sqlite                    3.42.0               h2c6b66d_0    conda-forge
tiledb                    2.13.2               hc2ae436_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
traitlets                 5.0.5                      py_0    conda-forge
tzcode                    2023c                h0b41bf4_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
wheel                     0.35.1             pyh9f0ad1d_0    conda-forge
xerces-c                  3.2.4                h55805fa_1    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.2                hfc55251_7    conda-forge

The command:

 result = subprocess.run(['pdal','translate','/dbfs/mnt/demo.obj','/dbfs/mnt/demo.las'],
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT )

returns:

CompletedProcess(args=['pdal', 'translate', '/dbfs/mnt/demo.obj', '/dbfs/mnt/demo.las', returncode=0, stdout=b'')

But the demo.las file is not created.

What was published to STDOUT? Any additional messages? Can you run your PDAL in --developer-debug mode? — Kyle Hale, Aug 02 '23 at 17:54
@KyleHale, i found the following: the output file is created if I ran the pdal with output file in /databricks location. But when i ran with a mounted location, the file is not created, and in log4j log, i find this: `23/08/07 20:15:15 ERROR WSFSDriverManager$: Failed to update command details for Files in Repos. Set Spark configuration value spark.databricks.enableWsfs=falseif you do not need the feature on this cluster. 23/08/07 20:15:15 ERROR WSFSDriverManager$: Files only supported within a repository. Invalid base notebook path: /data_ingestion/models_conversion/PDAL (1)` — stefania, Aug 08 '23 at 10:11

score 0 · Answer 1 · answered Aug 02 '23 at 20:47

Here is an example init script that has been used back on DBR 9.x to install and configure PDAL. This might be of use for your needs either as-is or you can incorporate into your docker build. Also, we offer GDAL (directly on DBR, not docker) which can be foundational for you to add PDAL [see docs].

dbutils.fs.put(
  "/init/install-pdal-DBR9.1.sh",
"""#!/bin/bash
phycores=$(cat /proc/cpuinfo|grep -m 1 "cpu cores"|awk '{print $ 4;}')
add-apt-repository ppa:ubuntugis/ppa
apt-get update
apt-get install -y gdal-bin libgdal-dev gcc-multilib

git clone https://github.com/hobu/laz-perf.git
cd laz-perf
git checkout tags/2.1.0
mkdir build
cd build
cmake ..
make -j $phycores
make install
cd /databricks/driver

git clone https://github.com/PDAL/PDAL.git pdal
cd pdal
mkdir build
cd build
cmake   -G "Unix Makefiles"  \
        -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_INSTALL_PREFIX=/usr \
        -DWITH_ICONV=ON \
        -DWITH_LASZIP=ON \
        -DWITH_LAZPERF=ON \
        -DWITH_LIBXML2=ON \
        -DBUILD_PLUGIN_PYTHON=ON \
        -DBUILD_PLUGIN_ICEBRIDGE=OFF \
        -DBUILD_PLUGIN_NITF=OFF \
        -DBUILD_PLUGIN_PGPOINTCLOUD=OFF \
        -DBUILD_PLUGIN_SQLITE=OFF \
        -DBUILD_PLUGIN_GREYHOUND=OFF \
        ..

make -j $phycores
sudo make install
/databricks/python3/bin/pip install PDAL
""", True)

i tried the script above, both on runtime 9.1 and 12.2 ML, but i am getting this error: _CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message): Could NOT find GDAL: Found unsuitable version "GDAL_VERSION-NOTFOUND", but required is at least "3.4" (found GDAL_LIBRARY-NOTFOUND) Call Stack (most recent call first): /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:391 (_FPHSA_FAILURE_MESSAGE) /usr/share/cmake-3.16/Modules/FindGDAL.cmake:153 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)_ Do you have a hint to pass this error? — stefania, Aug 03 '23 at 14:05

How to install PDAL library in databrisk cluster?

1 Answers1