0

I am running a hadoop mapreduce job using a Python program that creates different input paths as parameters for the mapreduce job. I am currently checking for hadoop fs path existence, before I pass these input paths into the mapreduce, using the command:

hadoop fs -test -e 'filename'

My Python program then communicates with the command line and determines if the file exists (the -test returns 0 when the file exists, an integer greater than 1 otherwise). Since the Python program is checking for path existence and outputting all of the nonexistent paths to a separate .txt document, I do not need to know which paths do not exist as command line warnings.

I would like to know how to suppress (or ignore) the automatic hadoop fs output:

test: 'fileName': No such file or directory

as I am inputting a huge number of paths and quite a few of them do not exist in hadoop fs.

matt123788
  • 419
  • 5
  • 16

1 Answers1

1

Redirect the error/warning to /dev/null/

hdfs dfs -test -e I/dont/exist 2>/dev/null
philantrovert
  • 9,904
  • 3
  • 37
  • 61
  • The command is fine. Also, last I checked, test only returns a number and doesn't print anything on the screen. Which version of hadoop are you using? – philantrovert Jun 13 '17 at 14:21
  • I am using hadoop 2.6.0 after a little more research I believe the output is actually from the Python package subprocess as I use that to call command line from within my Python class. This is where I'm at now: [Supress output from subprocess](https://stackoverflow.com/questions/7082623/suppress-output-from-subprocess-popen) – matt123788 Jun 13 '17 at 14:36
  • that looks like the right place to be since `hadoop test` doesn't print anything to stdout. – philantrovert Jun 13 '17 at 14:43
  • Thank you for your help @philantrovert, I do think your answer will help future users who have the question about `hadoop -test -e` output – matt123788 Jun 13 '17 at 14:49