Removing n columns from all the files from directory in Unix

Question

I have around 2000+ files which has random number of columns in each file. I wanted to remove last 4 columns from each of the file.

I tried to use below command, but it is not an inline command. Delimiter of the file is #

awk -F"#" '{NF-=4;OFS="#";print}' test > testing.csv

I wanted to save the file with the same name.(e.g. filename test with test only)

How to remove last 4 columns and save the file with same name? Can someone please help?

do you know the number of columns beforehand or does it vary? if you know the number, `cut` will be faster — Sundeep, Jun 02 '20 at 15:08
Does this answer your question? [Save modifications in place with awk](https://stackoverflow.com/questions/16529716/save-modifications-in-place-with-awk) — Quasímodo, Jun 02 '20 at 16:23

RavinderSingh13 · Answer 1 · 2020-06-02T16:48:29.497

3

In case you have GNU awk's latest version could you please try following.

gawk -i inplace -v INPLACE_SUFFIX=.bak 'BEGIN{FS=OFS=","} {NF-=4} 1' *.csv

This will take backup also for each csv Input_file.

Above is safe option which has your Input_file's backup too, in case you are happy with above command and DO NOT want backup files then you could simply run following.

gawk -i inplace 'BEGIN{FS=OFS=","} {NF-=4} 1' *.csv

NOTE: In case anyone using GNU awk version 5+ then we could use inplace::suffix='.bak' s per @Sundeep sir's comment here.

edited Jun 02 '20 at 16:48

answered Jun 02 '20 at 14:39

RavinderSingh13

130,504
14
57
93

1

note that on newer versions (I think 5+) you need to use `inplace::suffix='.bak'` ... also, OP is using `#` as delimiter – Sundeep Jun 02 '20 at 15:08
I have gawk 4.0.2. It does not have -i option to use. Could you please help? – Pranav Jun 02 '20 at 15:25
@Pranav, oh ok, then may you could William's answer which should work for you I believe. – RavinderSingh13 Jun 02 '20 at 16:15
@Sundeep, Thank you sir, added that in NOTE in my solution now, thank you for letting know here. – RavinderSingh13 Jun 02 '20 at 16:48

William Pursell · Accepted Answer · 2020-06-02T16:30:51.977

2

You really, really, really do not want to edit the files "in-place". It is (almost) always the wrong thing to do. For something like this, you want to do something like:

$ rm -rf new-dir/
$ mkdir new-dir
$ for file in old-dir/*; do 
    f=${file#old-dir/}; 
    awk '{NF-=4; $1=$1; print}' FS=# OFS=# "$file" > new-dir/"$f"; done

Then, after you know things have worked, you can replace your original directory with the new one.

edited Jun 02 '20 at 16:30

answered Jun 02 '20 at 15:03

William Pursell

204,365
48
270
300

While this is sound, I think it is overcomplicated. Can you please point to a reference that explains why inplace editing is generally wrong? – Quasímodo Jun 02 '20 at 16:25
1

Just worth mentioning that what `NF-=4` does is undefined behavior per POSIX so it could do anything in any given awk BUT luckily the OP is using GNU awk and (so far at least) that will remove the last 4 fields from each record. Quote your shell variables though! – Ed Morton Jun 02 '20 at 16:25
"in place" editing is overly complicated! The benefit of *not* editing "in place" is that it reduces the complexity. If you don't mind potential loss of data, then editing files "in place" is fine. Note that I consistently use double quotes around "in place" because it is almost always not actually editing the actual file, but creating a new file and overwriting it, but whatever utility you use is hiding that from you. – William Pursell Jun 02 '20 at 16:33
I agree, I prefer to have less things as possible hiding from me. But maybe worth putting that on your answer, just a suggestion :) – Quasímodo Jun 02 '20 at 16:37

score 2 · Answer 3 · answered Jun 02 '20 at 16:29

2

Using any POSIX awk:

tmp=$(mktemp) || exit 1
for file in *; do
    awk '{sub(/(#[^#]*){4}$/,"")}1' "$file" > "$tmp" &&
    mv -- "$tmp" "$file"
done

answered Jun 02 '20 at 16:29

Ed Morton

188,023
17
78
185

Removing n columns from all the files from directory in Unix

3 Answers3