2

Problem

I want to convert a string of paths (where each path may contain escaped spaces) into a printed list of paths. I'm mainly using echo and sed, and the problem lies therein (see below).

For example, I want these three files:

one two three\ two\ one

To be printed on three lines:

one
two
three two one

Problem with "sed" and "echo"

(1) These are three sample files: "one", "two", and "three two one".

echo "one two three\ two\ one"

# one two three\ two\ one

(2) I replace the whitespace that separates files with a newline.

echo "one two three\ two\ one" | sed 's/\([^\]\) /\1$'"'"'\\n'"'"'/g'

# one$'\n'two$'\n'three\ two\ one

(3) I test that indeed, "echoing" the output of the above results in the output I want.

echo one$'\n'two$'\n'three\ two\ one

# one
# two
# three two one

Combining (2) and (3), the functionality breaks using "xargs echo":

echo "one two three\ two\ one" | sed 's/\([^\]\) /\1$'"'"'\\n'"'"'/g' | xargs echo

# one$\ntwo$\nthree two one

How to fix the sed substitution?

How can the sed substitution be fixed so that echoing the output gives:

one
two
three two one

I'm looking for a solution that uses primarily "echo" and "sed" (not looking for other solutions).

I also can't use "echo -e" to interpret escaped characters.

Aeronautix
  • 301
  • 2
  • 13
  • 4
    This is very much an [XY problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). It's simple and easy to process arbitrary filenames correctly and robustly, but this question assumes the very bad starting point of a shell encoded string `one two three\ two\ one` instead of three files `one`, `two` and `three two one`. Can you take a step back? Where do you get filenames from, and what do you want to do with them? – that other guy Jun 01 '18 at 21:29
  • I get the filenames from finding them in a directory using a depth of 1: `find "$PWD" -maxdepth 1 | xargs -I abc sh -c 'echo $(printf %q "abc")'`, and I only want to display the filenames on the screen. I just want to know why the output of sed cannot be displayed to xargs echo in the same way as it does with simply echo. – Aeronautix Jun 01 '18 at 21:42
  • `printf %q` is a bash/ksh-only feature; it doesn't work with `/bin/sh`. – Charles Duffy Jun 01 '18 at 21:56
  • If you want to handle arbitrary filenames identified by `find`, use a NUL-delimited stream, or `-exec`. `find "$PWD" -maxdepth 1 -print0 | while IFS= read -r -d '' filename; do printf '%q\n' "$filename"; done` – Charles Duffy Jun 01 '18 at 21:57
  • ...or, for the latter: `find "$PWD" -maxdepth 1 -exec bash -c 'for arg; do printf "%q\n" "$arg"; done' _ {} +` – Charles Duffy Jun 01 '18 at 21:59
  • ...in no case is `xargs` (without `-0`) or `sed` part of a general-purpose or robust solution. – Charles Duffy Jun 01 '18 at 22:00
  • Also I want to add, the input can also come from something like brace expansion: `echo /usr/local/{lib/node*,node*,node_modules,node\\ with\\ spaces}`, as an example. – Aeronautix Jun 01 '18 at 22:12
  • Wait .. why doesn't `echo "one two three\ two\ one" | xargs -n 1 echo` work? Or perhaps `a=(one two three\ two\ one); printf '%s\n' "${a[@]}"`? I'm going to go with the XY Problem diagnosis. We can help you get past this hurdle, but what you REALLY need help with is your approach to the original problem. – ghoti Jun 02 '18 at 02:42
  • I'm trying to see why piping `$'\n'` from `sed` into `xargs echo` doesn't work. I will rephrase the problem as a simpler problem: (1) `echo "one two" | sed 's/ /$'"'"'\\n'"'"'/g'` gives `one$'\n'two` (2) `echo one$'\n'two` gives `one two` each on separate lines (3) `echo "one two" | sed 's/ /$'"'"'\\n'"'"'/g' | xargs echo` gives `one$\ntwo` (Problem:) Why doesn't it output `one two` each on separate lines? There's something fundamental going on, but what is it? – Aeronautix Jun 02 '18 at 20:03

3 Answers3

1

Here's how you show your filenames on screen separated by linefeeds:

find "$PWD" -maxdepth 1 -print

If you wanted to process them in some way, you should pass them \0 separated (using -print0) or use find -exec. Trying to format them as shell words is not the way to go.

The reason why echo doesn't show the value you expect is because $'\n' is Bash syntax meaning literal linefeed. It is not echo syntax, and echo is not involved in interpreting it.

For dash, xargs and other more traditional tools, $'\n' is just a weird way of formatting the literal characters dollar-backslash-n ($\n).

Since you're using xargs echo, bash is not involved, so you get $\n out instead. The solution is not to try to format data into code in such a way that the code will evaluate back to the original data. The solution is to skip that step entirely and just let data be data, such as in the initial example.

that other guy
  • 116,971
  • 11
  • 170
  • 194
0

This command works for me: 

echo "one two three\ two\ one" | sed 's/\([^\]\) /\1\n/g' | sed 's/\\ / /g' | xargs -L1 -I {} echo -{}-

The trick was

  • replace \\n with \n
  • use -L1 in xargs parameters
  • I used {} and -I{} just to demonstrate that echo is called for each line
Benjamin W.
  • 46,058
  • 19
  • 106
  • 116
Vicctor
  • 803
  • 6
  • 13
  • 1
    Works for that specific case. Doesn't work for the general case -- try putting some literal quotes in your filename, for example. – Charles Duffy Jun 01 '18 at 21:58
  • This gives me `-onentwonthree two one-`, I want the files output on newlines though. Is your machine getting the filenames all on separate lines? – Aeronautix Jun 01 '18 at 23:54
  • @Aeronautix - I'm using it on Ubutu18 and Bash from Cygwin on Windows. In both cases, the result is the as you written in the first post. What is your OS? Are you going to start it right from the command line, or it's a part of some bash script? – Vicctor Jun 02 '18 at 07:02
  • I'm running GNU bash 4.4 on macOS and the output is not on newlines, but when I run it on GNU bash 4.3 on Ubuntu 16, it works. Why? – Aeronautix Jun 06 '18 at 18:47
0

Not a solution but an answer to the comment you made on your own post "on why xargs echo" isn't working, see this answer regarding what $'\n' is:

https://stackoverflow.com/a/4128305/4092051

chipfall
  • 300
  • 2
  • 6