1

This will no doubt be simple for someone, but I'm not a bash wizard by any means. I tend to learn as I go when I need to do something. I've got a script that optimizes images and requires only that it be called with the image folder as an argument. It will determine which files can be used and travel down each subfolder as well.

I've created a simple loop that reads lines (paths to the image folders) from a txt file and feeds them to the optimization script one by one. My problem is that some of the folder trees contain a folder and subfolder branch that I want to skip, say a cache which contains a mirror of the image folder structure. What I need is a way to easily add that folder name, say to an exclude txt file, and have the bash loop ignore that and everything under it.

Here is the loop:

#!/bin/bash
filename='imgopt_sites.txt'

while read p; do 
    nice -n 10 bash /usr/local/bin/imgopt.sh $p > /dev/null 2>&1
done < $filename

Here is the txt file with folders to run on:

/srv/www/domain1.com/htdocs/wp-content/uploads
/srv/www/domain2.com/htdocs/wp-content/uploads
/srv/www/domain3.com/htdocs/wp-content/uploads
/srv/www/domain4.com/htdocs/wp-content/uploads
/srv/www/domain5.com/htdocs/wp-content/uploads
/srv/www/domain6.com/htdocs/image

An example of what would be excluded would be:

/srv/www/domain6.com/htdocs/image/cache (and all under that)

This could be in another txt file and would need to be looped as there will be more. Or, if there is an easy way to put them all in the same file and distinguish the exclude lines in some way that would be fine too. I've searched and nothing I've found really hits this. Thanks.

The OS is Ubuntu 14.04 and the loop is called by a cron job.

Last, here is a sample directory tree to give perspective:

/srv/www/domain6.com/htdocs/image
├── cache
│   ├── catalog
│   │   ├── apparel
│   │   ├── art
│   │   ├── banners
│   │   ├── commission
│   │   ├── demo
│   │   │   ├── banners
│   │   │   └── manufacturer
│   │   ├── illustrations
│   │   ├── jewelry
│   │   ├── misc
│   │   ├── news
│   │   ├── pen_ink
│   │   └── sample_data
│   │       ├── gallery
│   │       ├── logos
│   │       ├── patterns
│   │       ├── payments
│   │       └── slider
│   └── tb
├── catalog
│   ├── apparel
│   ├── art
│   ├── banners
│   ├── commission
│   ├── illustrations
│   ├── jewelry
│   ├── misc
│   ├── news
│   ├── pen_ink
│   └── sample_data
│       ├── banners
│       ├── features
│       ├── gallery
│       ├── logos
│       ├── patterns
│       │   └── _original
│       ├── payments
│       ├── slider
│       └── wallpaper
├── data
│   └── productimage
├── flags
└── templates

Any suggestions are appreciated.

Update: This is the section of code in the script itself that handles the input directories.

do_dir () {
  # $1 is name of dir
  local ret_val="0"
  while IFS= read -r -u 3 -d $'\0' file; do
    do_file "${file}"
    ret_val=$?
    done 3< <(find "${1}" -type f \( -iregex '.*\.[jp][pn]g$' -o -iregex '.*\.jpeg$' \) -print0)
  return $ret_val
}

I'm a little confused over what the "-u 3" does.

Update 2: New version of loop with errors in response to @shellter

#!/bin/bash

filename='imgopt_sites.txt'
excludes='imgopt_excludes.txt'

set -vx; while read p ; do nice -n 10 bash /usr/local/bin/imgopt/imgopt-ng.sh -new -progressive -sqlite $p; done <( grep -v -F -f "$excludes" "$filename" )

Error: 

/usr/local/bin/imgopt/imgopt_loop.sh: line 6: syntax error near unexpected token `<( grep -v -F -f "$excludes" "$filename" )'
/usr/local/bin/imgopt/imgopt_loop.sh: line 6: `set -vx; while read p ; do nice -n 10 bash /usr/local/bin/imgopt/imgopt-ng.sh -new -progressive -sqlite $p; done <( grep -v -F -f "$excludes" "$filename" )'
David Rahrer
  • 553
  • 1
  • 5
  • 11
  • 3
    maybe I"m missing a key detail in your description, but wouldn't `while read p ; do ..... ; done <( grep -v -F -f excludeListFile.txt "$filename" )` work? Good luck. – shellter Jul 21 '16 at 22:47
  • 1
    If `imgopt.sh` traverses the tree starting at the root passes as an argument is its responsibility to exclude whatever has to be excluded, not one of the calling shell. – Diego Torres Milano Jul 22 '16 at 03:05
  • Looks promising, thanks @shellter. When I put it in,however, I get the following error: `code` /usr/local/bin/imgopt/imgopt_loop.sh: line 7: syntax error near unexpected token `<( grep -v -F -f imgopt_excludes.txt "$filename" )' /usr/local/bin/imgopt/imgopt_loop.sh: line 7: `done <( grep -v -F -f imgopt_excludes.txt "$filename" )' – David Rahrer Jul 22 '16 at 03:14
  • @Diego Torres Milano I realize that would be proper, but the script is not mine and will probably be updated so I was hoping for a solution that kept separate from that. – David Rahrer Jul 22 '16 at 03:21
  • 1
    `read -u 3` meads _Read input from file descriptor fd._ Here fd is 3 which defined in your code `3< <(find "${1}" -type f \( -iregex '.*\.[jp][pn]g$' -o -iregex '.*\.jpeg$' \) -print0)`. In default `fd 1` is the stdout and `fd 2` is the stderr in shell. [man bash](http://linux.die.net/man/1/bash) – Ekeyme Mo Jul 22 '16 at 05:22
  • @DavidRahrer : Sorry, can't tell what is the problem in a comment. Maybe add another edit section at bottom of your Q and show `set -vx ; while read ... ; done <( ....)` block as it is executed and the error message that is generated. Good luck. – shellter Jul 22 '16 at 12:14
  • @shellter I've added the information you suggested in an update at the bottom of the OP. I've changed a couple of things but the error is the same either way. – David Rahrer Jul 22 '16 at 18:15
  • Assuming your script has `#!/bin/bash` at the top, you don't need to call `bash` to call your script. AND to eliminate another possible cause of problem, remove `nice -n 10 bash` from your loop. See if it works then, if so, then we can figure out what is not working. Good luck. – shellter Jul 22 '16 at 18:18
  • Best to dbl-quote all variable usage **all the time**, except for very rare special cases (which I can't remember right now ;-) ). so use `"$p"`. – shellter Jul 22 '16 at 18:20
  • weird, as your other code uses process redirection, i.e. `<(find ...)` . OK, thats all I got right now. Good luck. – shellter Jul 22 '16 at 18:21
  • @shellter I made the changes you suggested, and yes I have #!/bin/bash at the top so I removed bash. It gives exactly the same error. – David Rahrer Jul 22 '16 at 18:59
  • running out of ideas. Just incase the file has been MS-Windowized, check it with `cat -vet scriptName`. If you see any `^M` chars at the end of line, then run `dos2unix scriptName` and try again. Good luck. – shellter Jul 22 '16 at 19:31
  • @shellter Nope, all the lines end in $. I appreciate your help, I'm going to figure this out one way or another and I'll post the solution. – David Rahrer Jul 22 '16 at 19:54
  • 1
    well some final thoughts. 1. Given the error msg, confirm that a small test of `<( process_sub)` works. Make a separate script with the smallest test possible. (although it seems to work in the sub-script that you call). 2. Make sure the failing script is really running under bash. (not ash/dash/sh!). comment out the `<( ...)` in your problem script and add `echo $SHELL` just before the `while` loop. 3. (at the same time) You can replace the `nice -n 10 bash` with `echo "` (and a closing quote), and see if anything looks odd there. Good luck. – shellter Jul 22 '16 at 20:07
  • @shellter Well I figured out that error using your suggestion to isolate - the solution was to add another < between "done" and "<(..." i.e. done < <( grep...) Now the problem is that the original suggestion doesn't solve the original problem, lol. I end up with the path from imgopt_sites.txt being fed in and the script happily scans the cache folder under that and transverses all the others. I honestly think Diego Torres Milano was correct in that it will have to be done in the main script in the Find stuff. Thanks for the help, however, as I learned some things - always like that! – David Rahrer Jul 22 '16 at 21:17
  • Dang! I guess you need `< <(..)` when feeding a loop structure. Seems like you can still "anti-grep" paths that are too long (i.e. `.../cache`), but if you're sure, then I'll leave it to you. Best wishes and good luck. Post your answer and I'll vote for it! – shellter Jul 22 '16 at 23:17

0 Answers0