Using awk command to filter out specific columns from a huge file

Question

I'm currently working with a fairly large csv file that has 170 columns. I'd like to filter out three specific columns using awk filtering and then export that file out, so that the exported file only contains 3 out of 170 columns.

gzcat filename.csv.gz | awk '$19=="ACCOUNT_MOBILE_NUMBER" || $26=="DEVICE_ADID_TYPE" || $27=="DEVICE_ADID"' | gzip > filename_FILTERED.csv.gz

Expected results: new exported file containing only those three columns and their corresponding values/rows

Actual results so far: new exported file is blank

You need to provide a small snippet of your input file from `filename.csv.gz` — Inian, Jan 10 '19 at 04:48
It's a csv... so it's comma-delimited, no? Try your `awk` again with the `-F,` flag. — TrebledJ, Jan 10 '19 at 04:49
Your script looks for lines which contain one of those values in the respective columns (and would print all columns if it finds one). Evidently the file does not contain any lines with exactly those values. — tripleee, Jan 10 '19 at 05:26

score 0 · Accepted Answer · edited Jan 10 '19 at 05:25

0

I think you may be looking for something more along the lines of:

gzcat filename.csv.gz |
awk -F, '{print $19,$26,$27}' |
gzip > filename_FILTERED.csv.gz

edited Jan 10 '19 at 05:25

tripleee

175,061
34
275
318

answered Jan 10 '19 at 05:02

Sean Murphy

516
4
7

... Though if the CSV contains quoting, you might need something slightly more involved, perhaps vaguely like in https://stackoverflow.com/questions/52466382/how-do-i-extract-column-from-csv-with-quoted-commas-using-the-shell – tripleee Jan 10 '19 at 05:26
1

... And if it doesn't, `cut -d, -f19,26,27` is probably faster as well as more legible than an Awk script. – tripleee Jan 10 '19 at 05:28
cut -d function gave me what I needed. Thank you everyone – dfetap Jan 11 '19 at 06:11

Using awk command to filter out specific columns from a huge file

1 Answers1