0

I am writing a bash script that scans HDFS and do stuff with the output.

Fetching file names is easy with the following:

    for line in $(hdfs dfs -ls -t -r -C -R $HDFS_CLEANING_STG); do
        echo $line
    done

Output:

/dir
/dir/file1
/dir/file2

However, it removes the file size, date, rights, etc.

With the same method but without the -C flag, it gives the metadata but not on one line only:

Output example:

-rw-rw-r--+
3
hdfs
hdfs
34448169
2020-05-04
11:36
/dir/file

I would like to get these information, but with this output (like a "normal" ls):

-rw-rw-r--+ 3 hdfs hdfs 34448169 2020-05-04 11:36 /dir/file

How can I achieve that ?

Itération 122442
  • 2,644
  • 2
  • 27
  • 73

1 Answers1

2

The problem is in the for loop. When you do for line in $(hdfs dfs -ls -t -r -C -R $HDFS_CLEANING_STG); , it iterates over each word, not over each line. So, every time it faces a white space, the value of line changes with the word you just read.

A simple way to fix it is doing like this:

for line in "$(hdfs dfs -ls -t -r -C -R $HDFS_CLEANING_STG)"; do
    echo "$line"
done

Or like this:

hdfs dfs -ls -t -r -C -R $HDFS_CLEANING_STG | while read line; do
    echo "$line"
done
Francesco
  • 897
  • 8
  • 22