1

Objective

I'm trying to create a loop that will use my logins, select an account based on column 1and then select the platform based on column 2of input.csv. I have assigned each column to it's own variable in the hopes that it will loop through each row together and pull the reports i need, instead of me having to make a super long script of copy and paste.

input.csv

729352,7
355543,7
461432,7

Current Code

#! bin/sh

set -eu

value1=$(awk -F"," -v OFS=',' '{ print $1 }' input.csv)
value2=$(awk -F"," -v OFS=',' '{ print $2 }' input.csv)

rm -f cookiejar
curl  /dev/null -s -S -L -f -c cookiejar 'https://url/auth/authenticate' -d name=usr -d passwd='pwrd'

while IFS="," read -r value1 value2 remainder
do

curl -o /dev/null -s -S -L -f -b cookiejar -c cookiejar 'https://url/auth/adminaccounts' -d account=$value1
curl -s -S -L -f -O -J -b cookiejar -c cookiejar "https://url/report/ajax-by-tag2?platform_id[]=$value2&id1=&id2=&id3=&id4=&id5=&id11=&id12=&id13=&date=2019-11-25&date_start=&date_end=&website=&zfTablePage=1&zfTableColumn=&zfTableOrder=desc&zfTableQuickSearch=&zfTableItemPerPage=100&zfTableDataTablesMaxRows=1&zfDetails=false&by_viewability=imps_givt&device_id[]=all&tag_type_id[]=all&support_id[]=all&zfTableItemPerPage=10000&zfTableExport=xlsx"

done < "input.csv"

Current Error

+curl -o /dev/null -s -S -L -f -b cookiejar -c cookiejar https://urlauth/adminaccounts -d account=729352

&id1=&id2=&id3=&id4=&id5=&id11=&id12=&id13=&date=2019-11-25&date_start=&date_end=&website=&zfTablePage=1&zfTableColumn=&zfTableOrder=desc&zfTableQuickSearch=&zfTableItemPerPage=100&zfTableDataTablesMaxRows=1&zfDetails=false&by_viewability=imps_givt&device_id[]=all&tag_type_id[]=all&support_id[]=all&zfTableItemPerPage=10000&zfTableExport=xlsx

curl: (3) Illegal characters found in URL

What I have tried

  • I have replaced variable 2 in the url with the value "7" and can confirm that it loops through column 1 pulling each file.
  • I can confirm that variable 2 populates "7" correctly if I changed it with variable 1 (though it obviously doesn't download anything as it's not an account).
  • I've tried doing various different quote options around the url but every time it spits out the error above.
  • I've tried to put brackets around the variable {$value2}which I found worked when I used variables in the url previously.
  • Reviewing online, there is a lot of talk about the error being caused by different parts of the url, but I know the issue lies with the variable as it works without it. Can someone tell me what I seem to be missing to get this to work correctly?
El_Birdo
  • 315
  • 4
  • 19
  • Not related to your problem, but using `set -e` is... [not necessarily a good idea](http://mywiki.wooledge.org/BashFAQ/105#Exercises). (There are also [caveats to `set -u`](http://mywiki.wooledge.org/BashFAQ/112), but much more reasonable ones). – Charles Duffy Nov 26 '19 at 15:37
  • 1
    BTW, what's the purpose of the `awk` commands? Anything they put in `value1` or `value2` is replaced by `read` before the variable is ever used. – Charles Duffy Nov 26 '19 at 15:40
  • @CharlesDuffy the purpose of the awk commands is because I've only been teaching myself shell for the last few weeks and I couldn't think of a better way to do it XD. Just tried removing the `awk` and I can see it works without it. Thanks for the pointer! – El_Birdo Nov 26 '19 at 15:43
  • 1
    Ahh. FYI, a great resource is the Wooledge wiki; [BashFAQ #1](https://mywiki.wooledge.org/BashFAQ/001) goes into detail on correct usage of `while read`. – Charles Duffy Nov 26 '19 at 15:44
  • ...what I suspect is that your original file probably has DOS newlines, so the last field is always read as having a `$'\r'` on the end. That explains the log file starting a line in the middle (right after `value2`). – Charles Duffy Nov 26 '19 at 15:50

1 Answers1

0

It was due to not having a , after each line in the csv. So the input needed to be:

729352,7,
355543,7,
461432,7,
El_Birdo
  • 315
  • 4
  • 19
  • ...that said, I can't reproduce this behavior at all. `read -r a b r <<<'a,b'; echo "$?"` emits exit status `0`, meaning success, so the body would run. Can you speak to which specific shell you were testing with in producing this issue? – Charles Duffy Nov 26 '19 at 15:46
  • 1
    ...also, see https://ideone.com/u1X6rd, showing your original code working perfectly without needing the trailing commas to be added. – Charles Duffy Nov 26 '19 at 15:49
  • @CharlesDuffy I am using Ubuntu 18.04 on windows. Thanks for all the helpful documentation! – El_Birdo Nov 26 '19 at 15:50
  • "On Windows" is pertinent -- almost certainly, then, your input file is in Windows format and has CRLF newlines instead of LF-only newlines (as UNIX tools expect). One change you can make to work around that is to change `IFS=,` to `IFS=$',\r'`. – Charles Duffy Nov 26 '19 at 15:51
  • ...alternately, you could just use `dos2unix` to convert the file, or open it in vim, run `:set fileformat=unix` and save, etc. – Charles Duffy Nov 26 '19 at 15:52
  • ...anyhow, what adding the final `,`s is doing here is making the carriage return before the linefeed in the text file go into the `remainder` variable instead of being in `value2` where it gets passed to `curl`. – Charles Duffy Nov 26 '19 at 15:52
  • 1
    (BTW, see the *very first entry* in the "before asking about problematic code" section of https://stackoverflow.com/tags/bash/info). – Charles Duffy Nov 26 '19 at 15:53
  • @CharlesDuffy thanks for bringing this to my attention. I didn't realise there were these kinds of differences between the 2 systems. I'll aim to be more conscious of this in the future. – El_Birdo Nov 26 '19 at 16:10