Stack Overflow
Stack Overflow

Questions tagged [libhdfs]

Use this tag for questions relevant to libhdf; a JNI based C api for Hadoop's DFS.

20 questions
3
votes
2 answers

How to write on HDFS using pyarrow

I'm using python with pyarrow library and I'd like to write a pandas dataframe on HDFS. Here is the code I have import pandas as pd import pyarrow as pa fs = pa.hdfs.connect(namenode, port, username, kerb_ticket) df = pd.DataFrame(...) table =…
HHH
  • 6,085
  • 20
  • 92
  • 164
2
votes
1 answer

How do i set the path of libhdfs.so for pyarrow?

I'm trying to use pyarrow and i keep getting the following error. ImportError: Can not find the shared library: libhdfs3.so so i read some stackoverflow and it says that i need to set enviorment variable for ARROW_LIBHDFS_DIR. The path to…
Kush Singh
  • 157
  • 3
  • 11
1
vote
0 answers

Not able to connect to Hadoop Server using HadoopFileSystem in pyarrrow

I am trying a python code in which I am using pyarrow and trying to make connection to hadoop server using fs.HadoopFileSystem(host=host_value, port=port_value) but everytime I am getting an error message: self.parquet_writer =…
Aman Jain
  • 2,975
  • 1
  • 20
  • 35
1
vote
1 answer

Pyarrow fs.HadoopFileSytem reports unable to load libhdfs.so

I'm trying to use the pyarrow Filesystem interface with HDFS. I receive a libhdfs.so not found error when calling the fs.HadoopFileSystem constructor even though libhdfs.so is apparently at the indicated location. from pyarrow import fs hfs =…
cenrid
  • 11
  • 2
1
vote
1 answer

No such file exists while running Hadoop pipes using c++

While running hadoop map reduce program using hadoop pipes, the file which is present in the hdfs is not found by the map reduce. If the program is executed without hadoop pipes, the file is easily found by the libhdfs library but when running the…
samiullah ilyas
  • 87
  • 1
  • 8
1
vote
1 answer

ImportError: libarrow.so.14: cannot open shared object file: No such file or directory | python

I am getting below error when I am trying to install below library using File (.tar.bz2) . I dont have Internet connection in my hadoop cluster that is the reason I am using below command to install pyarrow using conda. (base) [XXXX@XXX-XXX-XXX…
gnish9
  • 537
  • 1
  • 5
  • 10
1
vote
0 answers

Segmentation Fault 11 when listing HDFS files

Apologies, because I don't know enough to ask this question correctly; all I know is that I'm getting a Segmentation Fault: 11 error whenever I try to list multiple files stored on HDFS using PyArrow with the libhdfs3 driver in Python3: Python 3.7.1…
Dan
  • 4,197
  • 6
  • 34
  • 52
1
vote
0 answers

How to resolve random crashes with Cloudera libhdfs.so?

Getting SIGSEGV crashes in libjvm.so when using Cloudera's libhdfs.so. These seem to occur at random points given the stack traces but most commonly during Java function Monitor::wait(). Any suggestions would be greatly…
jmay
  • 11
  • 3
1
vote
0 answers

ERROR : java.lang.IllegalMonitorStateException in c code using libhdfs

i tried to run program written in c++ code using hdfs c api. but it isn't working. this code written for read local file and write to hdfs like hdfs dfs -put command. here is my working environment : linux vesrion: 2.6.32-573.el6.x86_64 hadoop…
taraness
  • 11
  • 1
0
votes
0 answers

Pyarrow issue with hdfs connectivity

Earlier we were using pyarrow 0.13.0 version with python 3.7.16 version and through hdfs.connect we were able to establish connection with hdfs. As python version got upgraded to 3.9.10 we have to upgrade our pyarrow to 12.0.1 version ( hdfs.connect…
Abhay
  • 63
  • 1
  • 9
0
votes
0 answers

Pyarrow unable to load libhdfs on Windows 10

I am trying to use pyarrow on Windows but I'm getting the following error with fs.HadoopFileSystem() : OSError Traceback (most recent call last) Cell In[1], line 2 1 from pyarrow import fs ----> 2 hdfs =…
shane.singwa
  • 40
  • 4
0
votes
0 answers

Using libhdfs.so to perform hdfsConnect and Open File or Read leading to hdfsConnect call goes in to hung state

I am trying to create sample which uses client - server model to remotely deploy server component which uses libhdfs.so to perform hdfsConnect call which needs CLASSPATH and LIBHDFS_OPTS to be set to perform the sucessful connection and get hadoop…
Gajanan Kulkarni
  • 1
0
votes
1 answer

libhdfs - cannot open shared library libhdfs.so.0.0.0.0

I have a running hdfs instance, as evidenced by - reikdas@reikdas-HP-Pavilion-x360-Convertible-14-dh1xxx:~$ jps 16083 Jps 12389 NameNode 12774 SecondaryNameNode 11083 and reikdas@reikdas-HP-Pavilion-x360-Convertible-14-dh1xxx:~$ hadoop fs -ls…
Pratyush Das
  • 460
  • 7
  • 24
0
votes
0 answers

WinError 126 Error when connecting to HDFS using hdfs3

I am trying to read a file of a work HDFS location using the following code: import hdfs3 from hdfs3 import HDFileSystem hdfs=HDFileSystem(host='host',port='port') with hdfs.open('FILE') as f: model_AOB = f.read() I am getting the following…
TUIQ
  • 1
  • 1
0
votes
0 answers

When libhdfs hdfsRename will return -1

I use libhdfs in my c++ program, when I use hdfsRename to mv a file from old path to new path. the new path is not exist, and the old path is exist. But when I run the program, I get an error, the function hdfsRename just return -1 without any tips…
zawdd
  • 311
  • 1
  • 3
  • 12
1
2