Stack Overflow
Stack Overflow

Questions tagged [snakebite]

A pure python HDFS client.

17 questions
15
votes
6 answers

How to reset luigi task status?

Currently, I have a bunch of luigi tasks queued together, with a simple dependency chain( a -> b -> c -> d). d gets executed first, and a at the end. a is the task that gets triggered. All the targets except a return a luigi.LocalTarget() object…
HackToHell
  • 2,223
  • 5
  • 29
  • 44
3
votes
0 answers

Python API to get block locations of files in HDFS

I want to get the locations i.e., the Data Nodes where the blocks of a file are located in. I am using snakebite to perform administrative operations like snapshot and snapdiff. I am wondering if there are APIs in snakebite or any other python APIs…
rajath26
  • 95
  • 1
  • 1
  • 8
3
votes
1 answer

Configure SnakeBite HDFS clients to work with high availability mode

I'm using the snakebite library to access HDFS from my airflow dags. My HDFS cluster has been upgraded to High Availability Mode. This now means that clients configured to point to only one name node will fail when that namenode is not the active…
Marcin
  • 48,559
  • 18
  • 128
  • 201
2
votes
2 answers

HDFS: Read data from HDFS to parse XML files in HDFS using Python3

I have about 1500 XML files in HDFS, each of them is about 2-3Gb. I need to write a python script to parse the XML files to perform MapReduce. However, I am facing issue to access the files in HDFS using python. I tried the following script, and…
RRg
  • 123
  • 1
  • 12
2
votes
0 answers

Snakebite fails with Client mechanism is malformed

I am using the HDFSSensor in airflow, which somehow does not work, so I thought that there might be something wrong with snakebite. Therefore I tried running snakebite -D ls / in the terminal and it fails with the error Request error:…
Christopher Beck
  • 735
  • 8
  • 19
2
votes
1 answer

Python HDFS Snakebite : Methods work only with print

I am using the snakebite client from https://github.com/spotify/snakebite and i notice a strange behavior when i try to make a directory or move files around in hdfs. Here is my code. All it does it move the contents of the source directory to the…
AbtPst
  • 7,778
  • 17
  • 91
  • 172
0
votes
0 answers

How to connect to HDFS Cluster using snakebite-py3

I am trying to connect to an HDFS Cluster using python code, library(snakebite-py3) and I see that when I set use_sasl to True I am getting the following error: Code Snippet: from snakebite.client import Client client = Client(host='hostname',…
nikhil int
  • 181
  • 2
  • 14
0
votes
1 answer

Does snakebite handle retry in case of cluster failure?

snakebite recently come as an alternative to hdfs cli. According to my understanding hdfs does not do retry of command, if command failed due to cluster issue. My question here is that does snakebite handle the retry in case of cluster failure. By…
lifeisshubh
  • 513
  • 1
  • 5
  • 27
0
votes
1 answer

How to import "Security_pb2" package in Python 3?

I'm trying to import 'Security_pb2' package in python 3. But it's showing error. Can anyone tell me how to import it or any othet way to get it done. I even tried using to install it using "pip", but Could not find a version that satisfies the…
David
  • 366
  • 3
  • 22
0
votes
2 answers

How to import `snakebite` package in Python 3?

I'm trying to connect to Hadoop (HDFS) using Python. I'm using snakebite package for establishing connection. I installed it successfully, but when I'm trying to import it, it gives error as follows: from snakebite.client import Client client =…
David
  • 366
  • 3
  • 22
0
votes
2 answers

How to access kerberized cluster using snakebite python client

I have been struggling with how to pass requisite parameters to snakebite utility for it to be able to access a kerberized cluster. I have tried setting the necessary conf dir in the /usr/local/etc/hadoop path, as well as initialising and getting a…
Saurabh Mishra
  • 1,703
  • 3
  • 17
  • 27
0
votes
0 answers

ModuleNotFoundError in Using 3rd party module in external .py file

I installed third party module and its egg file was created in following path D:\Utkarsh\Lib\site-packages I am not getting error while importing module in IDLE in following way import snakebite When i am importing the same in HDFS.py file having…
Utkarsh Saraf
  • 475
  • 8
  • 31
0
votes
1 answer

Unable to fetch and display files using snakebite

I'm trying out snakebite. I started the following client: from snakebite.client import Client client = Client("my.host.com", 8020, effective_user='datascientist') First, I tried to list the users directory: for x in…
Dror
  • 12,174
  • 21
  • 90
  • 160
0
votes
1 answer

Snakebite HDFS touchz not working

I want to use snakebite to check if a file exist in hdfs directory, and create if it doesnt exist. I'm following the documentation on touchz here and using it like this: def createFile(client): if client.test("/user/test/sample.txt",…
DougKruger
  • 4,424
  • 14
  • 41
  • 62
0
votes
1 answer

Moving and merging directories in hdfs

I'm changing an hdfs directory structure. The current one is as…
TheSilence
  • 342
  • 1
  • 3
  • 11
1
2