1

I have list of file names file1,file2,file3
I want to pass these file names in the script and remove the special characters

I prepared sed command to remove the special characters 

sed  -i -e 's/^B/,/g' /home/data/nfiledata/ 
hdfs dfs -put -f /home/data/nfiledata/*  user/sai/table1/nfiledata/
gzip  /home/data/nfiledata/*

sed  -i -e 's/^B/,/g' /home/data/marginfile/  
hdfs dfs -put -f /home/data/marginfile/*  user/sai/table2/marginfile/
gzip  /home/data/marginfile/*

sed  -i -e 's/^B/,/g' /home/data/calldata/  
hdfs dfs -put -f /home/data/calldata/*  user/sai/table3/calldata/
gzip  /home/data/calldata/*

My question is instead of writing multiple times same command can i write in one command and loop the process for each file using Shell script 

nfile = (nfiledata,margindata, calldata)
while IFS= read -r nfile
do
  sed  -i -e 's//,/g' /home/data/$nfile/
  hdfs dfs -put -f /home/data/$nfile/*  user/sai/table$/$nfile/
  gzip  /home/data/$nfile/*
done < "home/data/$nfile"
Sai
  • 97
  • 1
  • 2
  • 17

2 Answers2

1

Answer for original version of question

The same sed command can be applied to edit-in-place several files with a single invocation:

sed  -i -e 's/old/new/g' /home/data/file1 /home/data/file2 /home/data/file3

Also, if the file names really are that simple, then brace expansion can be used:

sed  -i -e 's/old/new/g' /home/data/file{1..3}

Or,

sed  -i -e 's/old/new/g' /home/data/file[123]

Or, if there are no other similarly named files that you want to exclude, pathname expansion may be sufficient:

sed  -i -e 's/old/new/g' /home/data/file?

Example of real file names

sed  -i -e 's/old/new/g' nfile_dat fileidentifier margindata calldata
John1024
  • 109,961
  • 14
  • 137
  • 171
  • what about `/home/data/file[123]` or `/home/data/file?` ? – MaxU - stand with Ukraine May 26 '16 at 20:11
  • @MaxU Yes. I didn't show that originally because the OP's it is likely that `file1`, `file2,` etc are just stand-ins and the real file names are more complex. If not, however, I added your suggestions to the answer. – John1024 May 26 '16 at 20:16
  • Thank you for your answer John. What if my file names are different like (nfile_dat, fileidentifier,margindata,calldata) ? Also how can i loop the data – Sai May 26 '16 at 20:24
  • @Sai I just added to the answer an example using those file names. – John1024 May 26 '16 at 20:28
1

A for loop, not a while read loop, is appropriate here:

nfile=(file1 file2 file3)
for f in "${nfile[@]}"; do
  sed  -i -e 's/^B/,/g' /home/data/"$f"/ # should this be "$f"/* ?
  hdfs dfs -put -f /home/data/"$f"/*  user/sai/table1/"$f"/
  gzip /home/data/"$f"/*
done

Noteworthy components:

  • Assignments must not have spaces around the =. Commas are not part of array syntax in bash -- unquoted, unescaped whitespace acts as a separator in that case as elsewhere.
  • Expansions such as $f must be inside double quotes to be performed safely (without string-splitting or globbing).
  • Glob expansions, such as *, must be outside quotes to be honored.
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Thank you Charles. Even if my file names are different i can use this script? correct? My file names are like (nfile_dat, fileidentifier,margindata,calldata) . Each file represent a table in my hadoop. – Sai May 26 '16 at 20:28
  • 1
    @Sai, of course; put the actual directory names you want into the array contents. (Your code is written with `file1`, `file2` and `file3` being directory names, not file names; if they are in fact filenames, the original proposed commands wouldn't be correct either because of the trailing `/`s). – Charles Duffy May 26 '16 at 20:29
  • @Sai, ...so, if they're actually files, make it `/home/data/"$f"` where currently written `/home/data/"$f"/*` or `/home/data/"$f"/`. – Charles Duffy May 26 '16 at 20:30