Bash Issue : sed: -e expression #1, char 16: unknown option to `s'

Question

I'm trying pass a curl request and From response body , Im trying to grep for a key word "coin". If "coin" word matches , I need to get either "0" or "1" with the input URL as a prefix ( comma delineated ).

$for i in $(cat small);do curl -i -H "Accept: application/json" $i | grep "coin";echo $? | sed "s/^\(.\)/$i,\1/";done

#cat small

http://cuevana0.tv/
https://www.1111data.com/

I'm adding input as a prefix to the result "sed "s/^(.)/$i,\1/".

But i'm getting error as

$ for i in $(cat small);do curl -i -H "Accept: application/json" $i | grep "coin";echo $? | sed "s/^\(.\)/$i,\1/";done
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  633k    0  633k    0     0   909k      0 --:--:-- --:--:-- --:--:--  908k
    <script src="https://coin-hive.com/lib/coinhive.min.js"></script>
sed: -e expression #1, char 16: unknown option to `s'

Any suggestions on how to solve this ? Though the above is not a good practice , But it will help me timely.

Expected Result :

http://cuevana0.tv/,0

You need a compound statement to run multiple commands i.e. `.. | { grep "coin";echo $?; } | ..` and even with it, it is highly bad sequence of coding practices. Give us a requirement on what you clearly want to do with a sample i/p and expected o/p — Inian, Sep 29 '17 at 17:33
Thanks @Inian. I agree and I have updated the question above. — Arun, Sep 29 '17 at 17:40

Gordon Davisson · Answer 1 · 2017-09-29T17:58:26.830

The immediate problem is that you're using the "/" character as a delimiter in the sed expression, but the string you're substituting also contains "/", so sed mistakes that for a delimiter and gets very confused. For example, on the URL "http://cuevana0.tv/", you're running the command:

sed "s/^\(.\)/http://cuevana0.tv/,\1/"

...which is invalid sed syntax because of the extra "/" characters. You could switch to using a different delimiter, like ... | sed "s@^$.$@$i,\1@"; ... (provided you're sure "@" will never occur in one of your URLs).

But there's a much better way to do it. Rather than echoing the status and then using an edit to add the URL, just use echo "$i,$?".

Another thing here that's a bad idea: for i in $(cat small) is subject to a number of potential problems depending on what whitespace and/or wildcards the file might contain. In general, while read -r i; do ... done <small is much more predictable. Except that if anything inside the loop reads from stdin (or might read from stdin), you should use while read -r i <&3; do ... done 3<small instead.

Further explanation at http://mywiki.wooledge.org/DontReadLinesWithFor and http://mywiki.wooledge.org/BashFAQ/001 — glenn jackman, Sep 29 '17 at 18:43

Inian · Accepted Answer · 2017-09-29T18:45:48.440

I think a right way of coding this in bash avoiding useless cat usages and right usage of the tools (grep, return code) would be doing

#!/usr/local/env bash

while IFS= read -r url; do
    if curl -i -H "Accept: application/json" "$url" 2>&1 | grep -q "coin"; then
        printf "%s,0\n" "$url" 
    else
        printf "%s,1\n" "$url" 
    fi
done < urlFile

The above code uses right way to

Loop through a file using input file re-direction and using a while loop and the read command (avoids breaking up lines when it has spaces)
Using return code of grep directly in if clause by suppressing the search output -q to console
Passing both stdout and stderr of curl to grep to look for the keyword.
Double-quoting the variables to not let them split due to Word splitting when spaces are present.

As the great glenn-jackman points out, you could have the entire if-else clause replaced by this

curl -i -H "Accept: application/json" "$url" 2>&1 | grep -q "coin"; printf "%s,%d\n" "$url" $?

meaning to print the error code of grep right after the pipeline.

`curl -i -H "Accept: application/json" "$url" 2>&1 | grep -q "coin"; printf "%s,%d\n" "$url" $?` would be a little DRYer — glenn jackman, Sep 29 '17 at 18:42
@glennjackman I agree with the idea but I suspect `$?` might not be guaranteed to be `1` on a `curl` failure. We'd need to change `$?` to `$(( $? != 0 ))` or similar to be guaranteed to get a `0` or `1` value there. — Ed Morton, Sep 30 '17 at 14:28

Bash Issue : sed: -e expression #1, char 16: unknown option to `s'

2 Answers2