2

Im trying to curl a csv file and parse it based on their property and print it back by using variable names.

File 1:

10.0.0.1,gateway,name_of_device
10.2.4.5,server,name_of_device
10.3.5.6,PC,name_of_device

My script below,

#!/bin/sh

input=$(curl http://example.com/1.txt)

ip=$(echo "$input" | awk -F ',' '{print $1}')
type=$(echo "$input" | awk -F ',' '{print $2}')
name=$(echo "$input" | awk -F ',' '{print $3}')

echo "$ip $type $name"

This prints,

10.0.0.1
10.2.4.5
10.3.5.6
gateway
server
PC
name_of_device
name_of_device
name_of_device

but the expected output should be like,

10.0.0.1 gateway name_of_device
10.2.4.5 server name_of_device
10.3.5.6 PC name_of_device

Tried different options like,

assigning output to another variable and echo'ing it,

# output="$source_ip $tag_text"
# echo -e $output

printf statements:

# printf "%-20s | %-20s" "$source_ip" "$tag_text"


# printf "$source_ip" "$tag_text"
Karthik
  • 363
  • 2
  • 7
  • 20

2 Answers2

2

Just set the input field separator to the comma and let awk handle the rest of it:

$ awk -F, '$1=$1' file
10.0.0.1 gateway name_of_device
10.2.4.5 server name_of_device
10.3.5.6 PC name_of_device

By saying $1=$1, awk recalculates the fields and sets OFS all around, which makes all commas (FS) to be replaced with spaces (OFS).

fedorqui
  • 275,237
  • 103
  • 548
  • 598
  • You didn't `$1=$1`. That causes awk to rebuild the record and actually utilize the `OFS`. – James Brown Sep 02 '16 at 12:06
  • yes, this works. But how can i parse as rows? if I call an IP (first column), it should provide appropriate type and name (which is 2nd and 3rd column). – Karthik Sep 02 '16 at 12:19
  • @Karthik How do you "call an IP" ? Maybe using `grep` to retrieve the line with the specified IP is what you want? – Aaron Sep 02 '16 at 12:24
  • @Aaron may be yes. – Karthik Sep 02 '16 at 12:35
  • @fedorqui My actual requirement is I need to print certain columns alone, like IP and corresponding name_of_device. That's y i thought of assigning to a variable and print the required ones. Thanks. – Karthik Sep 02 '16 at 12:46
  • @Karthik then say for example `awk -F, '{print $2}' file` to print the `type`, for example, or `awk -F, '{print $2, $1}' file` to print type and then ip. – fedorqui Sep 02 '16 at 12:48
  • Yes got it, this works. but I will curl multiple csv files which may contain n number of columns, so I thought of assigning a meaningful name in the form of variable to print defined set of columns. I can see awk processes it very fast. – Karthik Sep 02 '16 at 12:52
  • @Karthik well, likewise you say `awk '{print $1}' file` you can say `awk -v my_col_number=5 '{print $my_col_number}' file` to print a specific position of a field. It really depends on what you want to do with this data; if it is just a matter of printing, stick to `awk`. If you need to do some processing with other tools, use a `while read` like the one in William Pursell's answer. But I cannot know what your final goal is if you don't provide more details (editing your question, preferably). – fedorqui Sep 02 '16 at 12:55
  • 1
    @fedorqui yes, understood. My final goal is to push these data into some db for other purposes. I really haven't started looking into db importing and it was too early to post that question here. So the db would contain a defied set of columns and that why i have to ignore them inside this script itself. Your suggestions was very helpful. thanks – Karthik Sep 02 '16 at 13:03
  • @Karthik OK, glad it was helpful :) In case you need to delete columns, take a look at [delete a column with awk or sed](http://stackoverflow.com/a/38145415/1983854). – fedorqui Sep 02 '16 at 13:06
2

awk is certainly the correct tool to use for this, but you can also do:

curl http://example.com/1.txt |
while IFS=, read ip type name rest; do
  echo $ip $type $name
done

The variables will lose their value after the while loop finishes, since it is in a sub-shell.

William Pursell
  • 204,365
  • 48
  • 270
  • 300
  • ... and if you want to have the variables in the main shell, use process substitution with `while ... done < <(curl ...)` – fedorqui Sep 02 '16 at 12:56