Create files matching a column value

Question

I have a text file similar to below

https://hosted.example.com [403]
https://checkout.example.com [200]
https://lib.example.com [403]
http://autodiscover.example.com [301]
https://go.example.com [503]
https://qa.example.com [403]
https://qalib.example.com [403]
https://join.example.com [200]

The file has similar lines with status codes in the second column like the above sample data. I want to create files based on the second column, if the second column has a value of 200 then that line has to be redirected to 200.txt and this follows with every status code.

Script used:

list=('[403] [401] [404] [503] [301] [302] [200]')
for i in $list
do
filename=$(echo $i | cut -d "[" -f2 | cut -d "]" -f1)
awk '$2 == "$i"' file.txt | awk '{print $1}' > $filename.txt
done

The problem with the script is it create files but the value of column 1 is not redirected to the respective files.

Ed Morton · Answer 1 · 2021-03-25T12:48:56.820

2

awk -F'[][]' '{close(out); out=$(NF-1)".txt"; sub(/ [^ ]+$/,""); print >> out}'

the above allows for [ and/or ] in the URL and won't fail with "too many open files" once you get past a limit with non-gawk awks.

With respect to '$2 == "$i"' in the script in your question - see How do I use shell variables in an awk script?.

edited Mar 25 '21 at 12:48

answered Mar 25 '21 at 12:31

Ed Morton

188,023
17
78
185

RavinderSingh13 · Answer 2 · 2021-03-25T12:51:17.893

1

With your shown samples, could you please try following.

awk '
/^https?/ && match($0,/\[[0-9]+\]/){
  outputFile=substr($0,RSTART+1,RLENGTH-2)".txt"
}    
NF{
  print >> (outputFile)
  close(outputFile)
}
' Input_file

edited Mar 25 '21 at 12:51

answered Mar 25 '21 at 08:30

RavinderSingh13

130,504
14
57
93

I tried and got `awk: null file name in print or getline input record number 1, file file.txt source line number 6` But why my code is not working ? – Joel Deleep Mar 25 '21 at 08:41
@JoelDeleep, Ok, could you please try my edited code and let me know if this helps you? – RavinderSingh13 Mar 25 '21 at 08:49
@JoelDeleep, This is a well tested command with your shown samples. Please do let me know if your actual Input is different from shown smaples? – RavinderSingh13 Mar 25 '21 at 09:07
there is no difference in the input – Joel Deleep Mar 25 '21 at 09:13
@JoelDeleep, ok, could you please do run a `cat -vet file` on your file to check if you have Control M characters in it? – RavinderSingh13 Mar 25 '21 at 09:20
`https://hosted.example.com [403]$` every line had $ at the end with that command. – Joel Deleep Mar 25 '21 at 09:26
1

A string like `[27]` can exist within a URL btw so YMMV with assuming it can't. – Ed Morton Mar 25 '21 at 12:47

Jetchisel · Answer 3 · 2021-03-25T10:50:51.807

1

This is the answer I have as far as I understood the question.

#!/usr/bin/env bash

while read -r col1 col2; do
  printf '%s %s\n' "$col1" "$col2" > "${col2//[][]}".txt
done < file.txt

That will overwrite the contents of the existing file if the value inside [ ] is a match, or rather if there are duplicate values.

edited Mar 25 '21 at 10:50

answered Mar 25 '21 at 09:37

Jetchisel

7,493
2
19
18

This worked, may I know why my code didnt work ? – Joel Deleep Mar 25 '21 at 09:38
You may paste your code at https://shellcheck.net for starters. – Jetchisel Mar 25 '21 at 09:39
1

The shell variable may or may not be interpreted by `awk` , also piping `awk` to `awk` is not really needed since awk can do it all. – Jetchisel Mar 25 '21 at 09:41

score 1 · Accepted Answer · answered Mar 25 '21 at 09:40

1

This awk should work for you:

awk '{gsub(/[][]/, "", $2); print $1 > ($2 ".txt")}' file

ls *.txt
200.txt  301.txt  403.txt  503.txt

answered Mar 25 '21 at 09:40

anubhava

761,203
64
569
643

1

This is a clean solution as always – Joel Deleep Mar 25 '21 at 09:44
1

@EdMorton there are no `[` & `]` in urls and as of now the file has only 7 unique status codes , which may rise but not above 8 or 10, so I dont think it will be an issue – Joel Deleep Mar 25 '21 at 15:08
1

@JoelDeleep you should add that information to your question. – Ed Morton Mar 25 '21 at 15:10

Create files matching a column value

4 Answers4