xargs does not read line by line properly

Question

Here is my current commands which i use:

xargs -n 1 -P 100 php b <links

I have a script 'b' and a file with links in 'links' and sometimes I don't know why this doesn't work correctly and add to every line symbol "?" like this:

root     19714  0.0  1.0  19880  5480 pts/2    R+   11:19   0:00 php b http://www.google.com?
root     19715  0.0  0.9  19524  4892 pts/2    R+   11:19   0:00 php b http://www.alexa.com?
root     19716  0.0  1.0  19880  5496 pts/2    R+   11:19   0:00 php b http://www.amazon.com?

see to the end of the ps-aux result line is "?" symbol and my file doesn't contain that... but not all the times... what could be the problem?

Could your `links` file have CRLF (WIndows or DOS style) line endings? In which case, the `?` represents the CR (carriage return)… And the fix is to convert the DOS file into a Unix file somewhere along the line: `tr -d '\r' < links | xargs -n 1 -P 100 php b`, for example. — Jonathan Leffler, Mar 03 '15 at 15:29
i build the links file with cat [another file] | sort | uniq > links ; So i don't think is a windows/dos style.... is just weird. — John Smith, Mar 03 '15 at 16:23
Does `[another file]` have DOS-format line endings? If so, `links` will too. Also, the use of `cat` is often unnecessary, and in this case `uniq` too is not needed; you could use `sort -u [another file] > links`. — Jonathan Leffler, Mar 03 '15 at 16:25
[another file] is like this: line1\r\n line2\r\n line3\r\n ..etc — John Smith, Mar 03 '15 at 16:27
The `\r\n` is one of the ways the CRLF line endings are reported on Unix. The `\r` notation is used for carriage return, CR, in C and other languages; the `\n` notation is used for newline (NL, aka line feed or LF). — Jonathan Leffler, Mar 03 '15 at 16:27
I will try in 30 minutes how you say, i will come back with an answer — John Smith, Mar 03 '15 at 16:28
BTW if i paste the same file with copy paste via vi or nano it works. — John Smith, Mar 03 '15 at 16:29
The editors may be converting the file format for you automatically. See: [How to convert the ^M linebreak to normal linebreak in a file opened in `vim`?](http://stackoverflow.com/questions/811193/how-to-convert-the-m-linebreak-to-normal-linebreak-in-a-file-opened-in-vim/811235#811235). — Jonathan Leffler, Mar 03 '15 at 16:30
i tried with tr -d... it works you can post as an answer and i can accept it. — John Smith, Mar 03 '15 at 16:36

score 2 · Accepted Answer · edited May 23 '17 at 12:20

^{Converting comments into an answer}

Could your links file have CRLF (Windows or DOS style) line endings? In which case, the ? represents the CR (carriage return)… And the fix is to convert the DOS file into a Unix file somewhere along the line. For example:

tr -d '\r' < links | xargs -n 1 -P 100 php b

I build the links file with cat "[another file]" | sort | uniq > links. So I don't think it is a Windows/DOS style issue.

Does [another file] have DOS-format line endings? If so, links will too; the command line will preserve the CRLF line endings if they're in the input.

Also, the use of cat is often unnecessary (UUoC or Useless Use of Cat), and in this case uniq too is not needed; you could use

sort -u "[another file]" > links

[another file] is like this:
line1\r\nline2\r\nline3\r\n

The \r\n is one of the ways the CRLF line endings are reported on Unix. The \r notation is used for carriage return, CR, in C and other languages; the \n notation is used for newline (NL, aka line feed or LF).

BTW: If I paste the same file with copy/paste via vi or nano it works.

The editors may be converting the file format for you automatically. See: How to convert the ^M linebreak to normal linebreak in a file opened in vim? Alternatively, if you are copying and pasting, then the copy may well not see, and therefore not copy, the CR characters.

I tried with tr -d … and it works.

Good; that at least means this is a reasonable explanation, and it all makes sense. If you don't need the list of links separately, you can use:

sort -u "[another file]" | tr -d '\r' | xargs -n 1 -P 100 php b

This does it all in one command line with no intermediate files to clean up. You could add tee links | before xargs if you do want the files list too, of course.

I forgot to say I filter the file with a php and contains \r\n(windows based) end lines... now i know there was the problem, should be only \n(unix based). — John Smith, Mar 03 '15 at 16:50

xargs does not read line by line properly

1 Answers1