0

I am trying to read a directory with "ls" and do operations on it

directory example:

$ ls -1
x x
y y
z z

script file: myScript.sh

#!/bin/bash
files=(`ls -1`); 
for ((i=0; i<"${#files[@]}"; i+=1 )); do
    echo "${files[$i]}"
done

however, the output is 

$ myScript.sh
x
x
y
y
z
z

yet if I define "files" in the following way 

$ files=("x x" "y y" "z z")
$ for ((i=0; i<"${#files[@]}"; i+=1 )); do echo "${files[$i]}"; done
x x
y y
z z

How can I preserve the spaces in "files=(`ls -1`)"?

Wayne Dawson
  • 31
  • 6
  • 3
    `for filename in *; do echo "$filename"; done`? Or `array=( * ); for ((i = 0; i < ${#array[@]}; i++)); do echo "${array[i]}"; done`? ***Never*** use `for i in $(ls anything)`, see [**Bash Pitfalls #1**](http://mywiki.wooledge.org/BashPitfalls#for_i_in_.24.28ls_.2A.mp3.29) – David C. Rankin Feb 12 '19 at 09:20
  • 4
    [Why *not* parse `ls`?](http://unix.stackexchange.com/questions/128985/why-not-parse-ls) – Cyrus Feb 12 '19 at 09:20
  • OK, aside from adding the $ in "${array[$i]}", both examples in 2 worked. I still don't understand why "ls" can be made to deliver a clear delineation between file names, but the operation (`ls -1`) destroys it, yet writing "*" works without any problems. Anyway, thank you for the advice, it was better than the solution I finally found. – Wayne Dawson Feb 12 '19 at 16:17
  • @DavidC.Rankin It seems the problem is that I shouldn't have used `ls`. Back in the old Unix days, that was generally how the books told you to do it. .. but I usually don't write files with white spaces either. – Wayne Dawson Feb 12 '19 at 17:12
  • Yes, that has turned into the evil scourge of shell scripting and has been properly banished in the name of portability between OS's where Mac has a history of doing really funky things with line-endings (like using a carriage-return alone in pre-OSX). I use sane filenames too and I don't embed newlines or carriage returns in them either, but the technically correct answers will accommodate the dingbat that does. Sigh, so we come kicking and screaming into the world of better scripting... – David C. Rankin Feb 12 '19 at 17:22
  • 1
    @DavidC.Rankin, ...bad filenames don't just come from random dingbats, they also come from bugs. You've heard my horror story about a buffer overflow in a Python library causing a Python script to create a filename that caused a shell script maintaining the same directory to delete `*`? Nasty things happen even in the absence of operator error. – Charles Duffy Feb 12 '19 at 18:12
  • 1
    Yes, @CharlesDuffy, that quip was meant more as a "tongue-in-cheek" bit of commiseration, rather than an all inclusive list of where nightmare filenames can come from. (I've even been the dingbat who on occasion has had a 3-liner go bad and and spew filenames I hard time getting rid of `:)` All in all, the lesson learned is to protect against the worst case rather than hope it doesn't exist on your box. – David C. Rankin Feb 12 '19 at 20:27

3 Answers3

4

Don't.

See:

If at all possible, use a shell glob instead.

That is to say:

files=( * )

If you need to represent filenames as a stream of text, use NUL delimiters.

That is to say, either:

printf '%s\0' *

or

find . -mindepth 1 -maxdepth 1 -print0

will emit a NUL-delimited string, which you can load into a shell array safely using (in modern bash 4.x):

readarray -d '' array < <(find . -mindepth 1 -maxdepth 1 -print0)

...or, to support bash 3.x:

array=( )
while IFS= read -r -d '' name; do
  array+=( "$name" )
done < <(find . -mindepth 1 -maxdepth 1 -print0)

In either of the above, that find command potentially being on the other side of a FIFO, network stream, or other remoting layer (assuming that there's some complexity of that sort stopping you from using a native shell glob).

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
1

It seems the main conclusion is not to use ls. Back in Pleistocene age of Unix programming, they used ls; however, these days, ls is best-restricted to producing human-readable displays only. A robust script for anything that can be thrown at your script (end lines, white spaces, Chinese characters mixed with Hebrew and French, or whatever), is best achieved by some form of globbing (as recommended by others here BashPitfalls).

#!/bin/bash
 for file in ./*; do
    [ -e "${file}" ] || continue
    # do some task, for example, test if it is a directory.
    if [ -d "${file}" ]; then
        echo "${file}"
    fi
done

The ./ is maybe not absolutely necessary, but it may help if the file begins with a "-", clarifying which file has the return line (or lines), and likely some other nasty buggers. This is also a useful template for specific files (.e.g, ./*.pdf). For example, suppose somehow the following files are in your directory: "-t" and "<CR>t". Then (revealing other issues with ls when using nonstandard characters)

$ ls
-t  ?t
$ for file in *; do ls "${file}"; done
-t  ?t
?t

whereas:

$ for file in ./*; do ls "${file}"; done
./-t
./?t

also

$ for file in ./*; do echo "${file}"; done
./-t
./
t

A workaround with POSIX commands can be achieved by --

$ for file in *; do ls -- "${file}"; done # work around
-t
?t
Wayne Dawson
  • 31
  • 6
  • BTW, if you only want directories, you can iterate over `./*/`. With respect to `./`, it doesn't make any difference for files that start with `*`s, but for files that start with `-`, the same issue can be worked around by using `--` as an end-of-options argument, as given in Guideline 10 of the POSIX utility syntax guidelines (http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html, seek to the bottom of the page). – Charles Duffy Feb 12 '19 at 20:18
  • @CharlesDuffy I fixed the "*" issue in the text. The original reason for the script was that I didn't notice that the default mask on Windows 10 Ubuntu is 000. So I was cleaning up the permissions on some SW I was developing. Anyway, thank you for your patience and advice. – Wayne Dawson Feb 14 '19 at 02:38
-3

Try this:

eval files=($(ls -Q))

Option -Q enables quoting of filenames. Option -1 is implied (not needed), if the output is not a tty.

Wiimm
  • 2,971
  • 1
  • 15
  • 25
  • 2
    This only adds overhead over `files=(*)` – tripleee Feb 12 '19 at 10:51
  • @Wiimm Interesting, I had tried `files=(\`ls -Q\`)`, but it failed. In the end, it seems the best solution seems to be just the brute force operator `*`. I never tended to use it because it seemed too unfiltered, but it seems to work, so that's what matters in the end. – Wayne Dawson Feb 12 '19 at 16:39
  • 1
    This doesn't work correctly with all possible filenames. `sp=' '; touch "${sp}${sp}${sp}"; eval files=($( ls -Q )); ls "${files[@]}"` will fail with `ls: cannot access ' ': No such file or directory`, to give you a concrete example. – Charles Duffy Feb 12 '19 at 17:43
  • 1
    ...and that's a *harmless* failure. `touch '$(touch evil)'` and then run `eval files=($( ls -Q ))` and you have a file named `evil` -- but you *could* have had something much more malicious if someone else had come up with that filename. – Charles Duffy Feb 12 '19 at 17:45
  • 2
    See [Why should eval be avoided in Bash, and what should I use instead?](https://stackoverflow.com/q/17529220/4154375). – pjh Feb 12 '19 at 17:49
  • 1
    Sorry you're getting down-voted to oblivion here, but please, please, **please** don't use `eval` when there's alternatives. – tadman Feb 12 '19 at 17:52
  • 1
    BTW, `eval "files=($(ls -Q))"` *with the quotes* would have avoided the less severe of the bugs I showcased above. Still depends on GNUisms and requires `ls` to generate correctly-escaped output for your specific shell, which can be iffy if it doesn't know about that shell's extensions. – Charles Duffy Feb 12 '19 at 17:52
  • 1
    ...the more severe bug (the security-impacting one) is caused by GNU ls (tested against coreutils 8.29) using double-quotes rather than single-quotes around a string that contains `$` in it. That's... pretty well inexcusable, but it does showcase the danger inherent in trusting tools to generate `eval`-safe data. – Charles Duffy Feb 12 '19 at 17:55
  • 1
    @WayneDawson, ...in terms of why ```files=(`ls -Q`)``` failed, see [BashFAQ #50](http://mywiki.wooledge.org/BashFAQ/050) -- the same issues (re: quotes not being treated as syntactic when present in an unquoted expansion's result) apply. – Charles Duffy Feb 12 '19 at 18:02