Alternative to xargs for processing pipeline

Question

I am trying to run a curl command which will need arguments coming in from a pipe

some operation | sort | head -n -2 | curl -s "blah/blah/<argument goes here>/blah"

i cant figure out how to pass each argument one at a time. One way that i found was

some operation | sort | head -n -2 | xargs -l "$" curl -s "blah/blah/$/blah"

but i get

xargs: $: No such file or directory

what does it mean? why is it trying to read $ as a file?

Not sure what were You trying to do. But Your first "$" is being interpreted as a command. And `-l` has been deprecated for some time. Basically, the answer would be --- because You just told to read $ as a file. — Kamiccolo, May 03 '16 at 19:08
`-l` takes a number, so `$` doesn't make sense as argument to it anyhow. — Charles Duffy, May 03 '16 at 19:20
well, echo is just an example. actually, i want to post a curl command for each argument that xargs receives — AbtPst, May 03 '16 at 19:25
...in general, though, when providing a command that's broken, it's helpful to describe what you *intend* it to do. After all, when reading it, we see the same thing that the OS itself does -- an invalid command; that doesn't tell us what was intended by its author. — Charles Duffy, May 03 '16 at 19:25
okay, you want to do a POST with `curl` -- with what characteristics around parallelization and error handling? As many as possible at once? One at a time? Exiting at the first error, or continuing? ...? — Charles Duffy, May 03 '16 at 19:27
...it may well be that `xargs` isn't the right tool for the job to begin with. I'd probably use a BashFAQ #1 loop if you don't need parallelization -- see http://mywiki.wooledge.org/BashFAQ/001 — Charles Duffy, May 03 '16 at 19:28
BTW, `-l` is a GNU extension -- it's not found in the POSIX specification for `xargs` -- so the generic unix tag (specifying a generic POSIX environment) isn't sufficient. — Charles Duffy, May 03 '16 at 19:31
so, the tricky thing here is that I can easily enough write an answer about the best-practices way to iterate over lines emitted from a pipeline (such as `sort | head`) and run an arbitrary shell command (such as `curl`) for each -- but that wouldn't be responsive to your question as asked, and would be unfair to the person who wrote the existing answer. — Charles Duffy, May 03 '16 at 19:33
...sure; the thing is, modifying the question in a way that makes prior answers invalid is frowned on because it wastes the work done by the people who answered the question in its prior state. — Charles Duffy, May 03 '16 at 19:36
makes sense, i did not know that xargs is not the proper way of doing this and i thought that maybe something is wrong with my environment. otherwise i would have framed the question differently — AbtPst, May 03 '16 at 19:38

Charles Duffy · Accepted Answer · 2016-05-03T19:42:54.637

4

See BashFAQ #1 for the best-practices way to read a string line-by-line:

#!/bin/bash
#      ^^^^ - not /bin/sh, or <() will be a syntax error

while IFS= read -r line; do
  curl -s "blah/blah/$line/blah"
done < <(some operation | sort | head -n -2)

edited May 03 '16 at 19:42

answered May 03 '16 at 19:38

Charles Duffy

280,126
43
390
441

thanks, but i get -r command not found – AbtPst May 03 '16 at 19:42
Can you pastebin *exactly* what you typed (or the contents of your script), how you invoke that script, and its output? Preferably somewhere without ads, ie. gist.github.com or ix.io – Charles Duffy May 03 '16 at 19:42
...if `-r` is being parsed as a command name, rather than as an argument to `read`, then the usage isn't a precise match for what I gave here. – Charles Duffy May 03 '16 at 19:43
please see the edit. i think the issue with r was improper spacing. i fixed that. now the $ is again being literally interpreted so my curl fails as it is an invalid parapmeter – AbtPst May 03 '16 at 19:47
@AbtPst, you notice that I used double-quotes, not single quotes? That's an important detail. If you want me to critique the correctness of the `curl` command line, I need to see the *actual* `curl` command line. – Charles Duffy May 03 '16 at 19:48
@AbtPst, ...also, note that I used `$line`, not `$` alone, to be replaced with the substituted contents. Please test that my answer works **exactly the way I gave it** before you go do something different and complain that that different thing didn't work. – Charles Duffy May 03 '16 at 19:49
@AbtPst, ...mind you, if what needs to follow the substituted *isn't* a `/`, then you may need to use `${line}`, not `$line`, to prevent extra characters to be treated as part of the variable. Once again, details matter, and if you hide the details of your work, we can't help find the problems in those details. – Charles Duffy May 03 '16 at 19:51
perfect, it works now. i had put $ instead of $line. so in this case, are $line and ${line} equivalent? – AbtPst May 03 '16 at 19:53
`${line}` and `$line` are equivalent except when you need to concatenate with a literal string that can be part of a name. For instance, `$line_foo` tries to find a variable named `line_foo`, whereas `${line}_foo` expands the variable named `line`, and puts `_foo` on the end of the result. – Charles Duffy May 03 '16 at 19:54
got it. i tried to put ${line} at the end as `curl -s "$CURL_URL/GET/${line}"` but now the argument gets passed in as `[xyz-*]` which is invalid. how can i correct this? – AbtPst May 03 '16 at 19:57
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/110937/discussion-between-charles-duffy-and-abtpst). – Charles Duffy May 03 '16 at 20:01
1

unfortunately, i cant access the chat feature from work. is there a quick fix for this? i really appreciate your help – AbtPst May 03 '16 at 20:02
First place to start is to compare the command line emitted by `bash -x yourscript` with the curl command line you'd write by hand for that same URL. Once you've done that, you have a better question -- albeit one that should be asked *as a separate question*, if it's not already in the knowledgebase. – Charles Duffy May 03 '16 at 20:06
i did `echo "curl -s \"$CURL_URL/GET/${line}\""` and i get `curl -s "http://hostName:port/GET/xyz123"` which is perfect. so, now why does this command not get executed? from the error i get, it looks like the command that is being executed is `curl -s "http://hostName:port/GET/[xyz123]"`. how is it possible? – AbtPst May 03 '16 at 20:09
Don't ever trust echo. Use `set -x` (inside the script) or `bash -x yourscript` (when invoking it), as I said. – Charles Duffy May 03 '16 at 20:12
...in particular, `\"...\"` is wrong; that makes *literal* quotes, not *syntactic* quotes, which are the kind you need. – Charles Duffy May 03 '16 at 20:13
...`echo` prints only literal quotes, which is part of why believing that what `echo` emits is equivalent to what the same command without `echo` on the front runs is so very wrong. – Charles Duffy May 03 '16 at 20:14
i did bash-x and i got `curl -s http://hostName:port/GET/xyz123 {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No feature for name [xyz123]"}],"type":"illegal_argument_exception","reason":"No feature for name [xyz123]"},"status":400}` so i cant understand whats going on. – AbtPst May 03 '16 at 20:24
If the `bash -x` doesn't show `[xyz123]` in the argument list, then that's not what is actually being run. – Charles Duffy May 03 '16 at 20:33

Alternative to xargs for processing pipeline

1 Answers1