How to modify multiple files using a bash script

Question

I have a file called script.sh with the following code:

#!/bin/bash

file=list.txt
IFS=$','
cat $file 
for i in `cat $file`
do
    echo "The name of the file I am going edit is $i"
    awk 'BEGIN{FS=OFS=","} /^\<AA_xx\>/||/^\<AA\>/{$160="CC,0,DD,0"} 1' $i \
        > temp_file.csv && mv temp_file.csv $i
done

list.txt contains the path to a .csv file, e.g., /ws/user/stack.csv.

However, I would like to modify this script to read all the files I specify in list.txt, i.e., my list.txt contains

/ws/user/stack.csv
/ws/user/stack1.csv
/ws/user/stack2.csv

I want the script to use this awk in each file mentioned in list.txt.

Can you help me with this?

What are you saying is the issue here? Are you saying lilst.txt isn't being read properly? I you set `IFS=","`. Are the filenames in list.txt comma separated or newline separated? If newline separated, checkout out https://stackoverflow.com/questions/10929453/read-a-file-line-by-line-assigning-the-value-to-a-variable — Tyler Marshall, Dec 06 '18 at 16:28

rvbarreto · Answer 1 · 2018-12-08T11:17:35.703

0

The following does the trick

script.sh

#!/bin/bash
while read -r line; do
    cat "$line" | awk '{print $0}' > "output_$line"
done < "$1"

Then you just call it with

$ ./script.sh list.txt

If you have a large awk code, I recommend organizing it in a function.

script.sh

#!/bin/bash
yourAwkFunction(){
    awk '{print $0}'
}

while read -r line; do
    cat "$line" | yourAwkFunction > "output_$line"
done < "$1"

line is the variable that receives the value of each line of list.txt. This means that line is /ws/user/stack.csv for the first iteration and then /ws/user/stack.csv1 for the second and so on.

For more on reading lines of files, you may check http://mywiki.wooledge.org/BashFAQ/001

edited Dec 08 '18 at 11:17

answered Dec 06 '18 at 16:32

rvbarreto

683
9
24

Does this take care of new line separation of each file? I donot need IFS right? – ShivaniSarin Dec 06 '18 at 16:44
I've tested with a list.txt in which each line corresponds to a file name and it worked. If you have a newline separated list.txt, you don't need to specify IFS – rvbarreto Dec 06 '18 at 16:48
what is line in this case and $line? – ShivaniSarin Dec 06 '18 at 16:56
I've clarified in the answer – rvbarreto Dec 06 '18 at 17:01
how do I modify my awk function here? $i \ > temp_file.csv && mv temp_file.csv $i What is $i in this case then? – ShivaniSarin Dec 06 '18 at 17:27
I've changed the code. Just copy and past and it will do what you want – rvbarreto Dec 06 '18 at 19:39

score 0 · Answer 2 · answered Dec 06 '18 at 16:51

0

Since your comment says list.txt is newline separated, just follow how to read a file line by line here: Read a file line by line assigning the value to a variable

Which would give you:

#!/bin/bash

while IFS='' read -r file || [[ -n "$file" ]]; do
  echo "The name of the file I am going edit is $file"
  awk 'BEGIN{FS=OFS=","} /^\<AA_xx\>/||/^\<AA\>/{$160="CC,0,DD,0"} 1' $file \
    > temp_file.csv && mv temp_file.csv $file
done < list.txt

answered Dec 06 '18 at 16:51

Tyler Marshall

489
3
9

How do I run the script? just ./script.sh? – ShivaniSarin Dec 06 '18 at 16:58
Exactly. Here, I just hardcoded list.txt as input. You can also use `"$1"` in place of list.txt above and run with `./script.sh list.txt` – Tyler Marshall Dec 06 '18 at 16:59
The name of the file I am going edit is /ws/user/stack.csv The name of the file I am going edit is /ws/user/stack1.csv The name of the file I am going edit is mv: missing destination file operand after `temp_file.csv' Try `mv --help' for more information. – ShivaniSarin Dec 06 '18 at 17:22
The third line the name of the file I am going to edit is is empty.. and then it says missing destination file from operand – ShivaniSarin Dec 06 '18 at 17:23
This is because it is trying to mv temp.csv to /ws/user/stack.csv .. which is an invalid location. How do I go to one directory above – ShivaniSarin Dec 06 '18 at 17:48
How is /ws/user/stack.csv an invalid location? Either way, if you want to go one directory use `..`. So saying `/ws/user/../` would be the directory `/ws/user/`. – Tyler Marshall Dec 06 '18 at 21:33

How to modify multiple files using a bash script

2 Answers2