1

I keep getting bitten over and over again with newline issues.

My first issue is, I have a string in bash where I'm appending "\n" via the echo -e command, because I want each result to be on it's own line, as I'm going to output that string eventually to file. When I echo the string out, it looks correct. If I grep through it trying to find one line,

var=$(echo $var2 | grep pattern)

I discover that it sees the string as one line. I don't want that. How do I change that so grep would work as expected, matching the lines of interest (question 1).

Also, if you have any insights on the following, seemingly related, I'd appreciate it: when I do a find on the command line, it separates results by (apparently) newlines. Assigning the results of find to a bash variable and then grepping or searching through that, it appears the find results are all one line. I've tried/researched some of the

find -print

options but haven't been successful, perhaps it is a similar problem to number 1. I'm not getting something here. The second question would be, what's the right option to use on a find command so when I assign it to a variable and grep through the results, it works like it would on the command line (meaning find | grep).

Thanks.

that other guy
  • 116,971
  • 11
  • 170
  • 194
Ray
  • 5,885
  • 16
  • 61
  • 97
  • Can you include a reproducer? Without one, it's not clear what you mean. – Charles Duffy Aug 18 '15 at 22:27
  • I strongly suspect that somewhere, you're doing an `echo $string` rather than a `echo "$string"`. – Charles Duffy Aug 18 '15 at 22:30
  • ...also, don't use `echo -e`. See http://pubs.opengroup.org/onlinepubs/009604599/utilities/echo.html, particularly APPLICATION USAGE and RATIONALE sections: The POSIX specification for `echo` doesn't allow an implementation passed `-e` to do anything but print that `-e` on its output; thus, you're explicitly relying on having an implementation that breaks the standard. – Charles Duffy Aug 18 '15 at 22:31
  • Charles, you were right. I had a var=$(echo $var2 | grep pattern). I left off the quotes. – Ray Aug 18 '15 at 22:33
  • Consider `var=$(grep pattern <<<"$var2")` instead, btw. No need for a pipeline there. – Charles Duffy Aug 18 '15 at 22:34
  • Also, I strongly advise against using `find -print` programatically -- keep in mind that newlines are valid characters inside filenames, so you can't safely use them to *separate* filenames. – Charles Duffy Aug 18 '15 at 22:35
  • Ok. Per the "if you must", should I not use grep? – Ray Aug 18 '15 at 22:35
  • Depends on the context. Grep is often an appropriate tool. – Charles Duffy Aug 18 '15 at 22:35
  • ...on the other hand, grepping the output of `find` is often evil. – Charles Duffy Aug 18 '15 at 22:36
  • Thanks. What's a print option on find, that when assigned to a bash variable, it's going to properly get newlines in the bash variable? Just for those odd cases. – Ray Aug 18 '15 at 22:36
  • Well, that's the thing -- you don't *want* newlines in the bash variable unless the filename itself contains newlines. Instead, you should use NUL-delimited output, and load content into an array. – Charles Duffy Aug 18 '15 at 22:37
  • And then what do I use to search the array (vs) grep, and say feed (pipe) the result into some other command? – Ray Aug 18 '15 at 22:38
  • `files=( ); while IFS= read -r -d '' filename; do files+=( "$filename" ); done < <(find . -print0)` gives you an array called `files`, which you can then print unambiguously with `printf '%q\n' "${files[@]}"`, enumerate the size of with `echo "Found ${#files[@]} files"`, or whatnot. – Charles Duffy Aug 18 '15 at 22:38
  • If you want to search through the list of filenames, you should be telling `find` to filter it. It's very, very flexible on that count. – Charles Duffy Aug 18 '15 at 22:38
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/87323/discussion-between-charles-duffy-and-ray). – Charles Duffy Aug 18 '15 at 22:38
  • I want all the files. I just need to do different operations on the results that I don't want to do in an exec (for now). – Ray Aug 18 '15 at 22:39
  • Can't, my company firewall won't allow a chat. – Ray Aug 18 '15 at 22:39
  • So, if I can, if I put the results into an array, how do I find "abc" in that array and pipe the result to some other command? Thanks for your help BTW. – Ray Aug 18 '15 at 22:41
  • If you're trying to find something in the array, you should have been telling `find` to filter it earlier. – Charles Duffy Aug 18 '15 at 22:43
  • Maybe that means you should have used `-name '*abc*'`, or `-regex '.*abc.*'`. – Charles Duffy Aug 18 '15 at 22:43
  • I want broad find results to do one thing, and then further filter the results and do other things. – Ray Aug 18 '15 at 22:44
  • Can't give good situational advice based on broad descriptions. Details always matter. – Charles Duffy Aug 18 '15 at 22:44
  • That said, you can emit an array as a NUL-delimited stream again with `printf '%s\0' "${array[@]}"`, and filter through that list with `grep -z` (if you're on a platform with GNU tools). – Charles Duffy Aug 18 '15 at 22:45
  • ...modern systems will generally have tools that can deal with NUL-delimited streams (grep -z, xargs -0, find -print0, etc); failing to cover this was one of the bigger design failures in the POSIX spec. – Charles Duffy Aug 18 '15 at 22:48
  • Thanks, (again, I know general), is grep a good/efficient tool to use to find a pattern in a bash array (as you showed), or should I use a bash array search mechanism and pass that on? – Ray Aug 18 '15 at 22:48
  • ...and of course you can iterate through and filter bash arrays in native bash as well. – Charles Duffy Aug 18 '15 at 22:48
  • It's all situational. If you're going to be doing lots of searches for specific item rather than a substring, the right thing to do is to load your items as keys in an associative array (what other languages might call a hash or a map), so your lookups for them are O(1). – Charles Duffy Aug 18 '15 at 22:49
  • Suggest you might consider starting with http://mywiki.wooledge.org/BashFAQ/005, by the way. – Charles Duffy Aug 18 '15 at 22:50
  • Thanks Charles, I see. Often I need to find substrings, so thanks for differentiating that and what to read. – Ray Aug 18 '15 at 22:52
  • Usually, though, I tend to avoid storing such lists at all -- iterating over contents in realtime (see FAQ 1) is considerably more memory efficient than storing for later processing. – Charles Duffy Aug 18 '15 at 22:54
  • ...and it's easy to do pattern matches during real-time processing: `while IFS= read -r -d '' filename; do case $filename in *abc*) : "do something";; *def*) : "do something else";; esac; done < <(find . -print0)` – Charles Duffy Aug 18 '15 at 22:55
  • Thank you. I need to make more time to study Bash rather than just trying to get the job done as fast as possible. I appreciate your support. – Ray Aug 19 '15 at 16:36

0 Answers0