Hadoop hdfs shell command for inverse pattern search

Question

I would like to know if we can list files in an HDFS directory with files NOT matching the given pattern.

For instance, assume I have some files with names containing some date values [eg: 'file201606250001.csv'].

If I need to get list of files not belonging to a date pattern, say '*20160629*', what command can I use?

I tried to give the following command, but it still is listing all files (including the ones matching the pattern I wanted to exclude).

hdfs dfs -ls /test/* | grep -v '*20160629*'

I tried the solution given here, but still it returns all files.

Possible duplicate of [List files not matching a pattern?](http://stackoverflow.com/questions/8525437/list-files-not-matching-a-pattern) — Inian, Jun 29 '16 at 12:43
@Inian I tried that solution, but it is not working in my case. It still lists files with the pattern to be excluded. — Hrishikesh, Jun 30 '16 at 06:57

score 1 · Accepted Answer · answered Jun 29 '16 at 13:22

1

First list the file and then grep other than your pattern (-v option)

hadoop fs -ls /dir/path/to/search |grep -v '20160629'

answered Jun 29 '16 at 13:22

Ronak Patel

That worked. I was giving `grep -v '*20160629*'` after the pipe as the command. Thanks! – Hrishikesh Jun 30 '16 at 07:03

1 Answers1