0

I'm creating a function to perform a full-system backup using rsync on my various devices. In order to troubleshoot, I've enabled set -x to see what the command is actually doing. There appear to be extraneous single quotes surrounding the --exclude={...} portion which is preventing it from working properly. Any ideas what might be causing this?

My function:

function system-backup {
    u_excl=(\"/dev/*\", \"/proc/*\", \"/sys/*\", \"/tmp/*\", \"/run/*\", \"/mnt/*\", \"/media/*\", \"/lost+found\")
    source=$HOSTNAME
    sourcedir='/'
    dest='192.168.1.51:'

    # determine additional folders to exclude and other options
    if [ $HOSTNAME == my-pc ]; then
            u_excl+=(,\"/home/*\")
    fi

    excl=$(printf {"%s" "${u_excl[@]}"})

    ( set -x; sudo rsync -aAXHv --delete --exclude={"$excl"} "$sourcedir" "$dest"'/mnt/external/bkps/'"$source"'_bkp' )
}
export -f system-backup

And my output:

$ system-backup
+ sudo rsync -aAXHv '--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/home/*"}' / 192.168.1.51:/mnt/external/bkps/my-pc_bkp
meijin3
  • 1
  • 2
  • 1
    Put your code in https://www.shellcheck.net/, it will point out some elements that might help. – Nic3500 Aug 16 '22 at 15:06
  • 1
    `set -x` displays commands in the way you would type them into the shell. The apostrophes you see are not part of the argument; they are necessary because the double quotes in the argument would otherwise be removed. It's the double quotes which are wrong, but you carefully entered them, using `\"` to ensure that they form part of the pattern. Since there are no quotes in the filepath, that's not going to match anything. But that's just one of the problems. You cannot specify multiple patterns in an `--exclude` option. Commas are not used between array elements in bash. – rici Aug 16 '22 at 15:27
  • As rici says, the right way to define your array would like `u_excl=( '/dev/*' '/proc/*' '/sys/*' '/tmp/*' '/run/*' ... )` -- though it's very silly to do that when you can just tell rsync not to cross mount points and avoid having any reason to need to list them out in the first place. Similarly, `u_excl+=( '/home/*' )` is the correct extension syntax, and `"${u_excl[@]}"` is the correct expansion syntax. – Charles Duffy Aug 16 '22 at 15:31
  • The `excl=$(printf ...)` is just wrong on its face, to the point that I can't tell what you _meant_ it to do so I can't tell you how to correct it. Converting your array into a string before substituting it is the exact opposite of what you _should_ be doing. (For more on "what you should be doing", see [BashFAQ #50](https://mywiki.wooledge.org/BashFAQ/050), particularly when it gets into the "what do I do instead?" section). – Charles Duffy Aug 16 '22 at 15:32
  • @rici Is it possible that `rsync` has a non-standard syntax? I've followed the example I initially found on the Arch wiki and it has always worked for me: https://wiki.archlinux.org/title/Rsync#Full_system_backup In other words if I type the following command (note the lack of single quotes), it works exactly as expected: `sudo rsync -aAXHv --exclude={"/dev/*"} / 192.168.1.51:/mnt/external/bkps/my-pc_bkp` – meijin3 Aug 16 '22 at 15:34
  • Now, if you want a separate `--exclude=` argument for each element in `u_excl`, then maybe you want `"${u_excl[@]/#/--exclude=}"` – Charles Duffy Aug 16 '22 at 15:35
  • @meijin3, this is all bash syntax; the parsing happens before rsync even starts. rsync has no control over how the shell does its end of command-line processing. – Charles Duffy Aug 16 '22 at 15:36
  • You need to try to understand how the shell processes what you type. Try looking at the output of `echo rsync ...`. Then do it again with `set -x` enabled so that you can see how `set -x` shows you a command line – rici Aug 16 '22 at 15:38
  • @meijin3, similarly, there are lots of different bash commands that all evaluate to _the exact same_ C string array passed as an argv. `rsync "foo"` and `rsync 'foo'` and `rsync foo` are three ways of writing _the exact same command_, all three of which become (in C syntax) `char[][]{"rsync", "foo", NULL}`. By "the exact same", I mean that `rsync` itself can't even tell which one of them was used. – Charles Duffy Aug 16 '22 at 15:38
  • @CharlesDuffy The reason why I've converted my array into a string in that way is because in the list of exclusions, I can't have any spaces. If you know of a way way to do this better, please let me know. Although now I'm thinking that an array is not the tool for this job and I should just be concatenating strings. – meijin3 Aug 16 '22 at 15:39
  • I would actually suggest instead of `echo rsync ...`, testing with `printf '%s\n' rsync ...` -- there are places where echo makes things look the same when they're really not. (For example, you can't tell the difference between `echo hello world` and `echo "hello world"`, even though they're very different). – Charles Duffy Aug 16 '22 at 15:39
  • @meijin3, no, the array is **absolutely** the right tool for the job. Follow the BashFAQ #50 link I gave you earlier. – Charles Duffy Aug 16 '22 at 15:39
  • @charles: `printf` is good, too. But `set -x` shows what is going on, and it's important that OP understands how to read it. – rici Aug 16 '22 at 15:44
  • @meijin3, see the online sandbox at https://ideone.com/ylQyS2 running a version of your code corrected per the instructions I gave earlier (`sudo` is changed to `:` to make it a noop; you can change it back for your real program). – Charles Duffy Aug 16 '22 at 15:45
  • @meijin3, ...so, I just realized an aspect of the question that I think you didn't spell out, and consequently we didn't answer. Brace expansion -- replacing `--exclude={foo,bar,baz}` with `--exclude=foo` `--exclude=bar` `--exclude=baz` -- is one of the first steps bash does; it's done **before** variables are expanded, so you can't use the result of variable expansion to perform brace expansion. One of the simple cases of that is covered in [BashPitfalls #33](https://mywiki.wooledge.org/BashPitfalls#for_i_in_.7B1...24n.7D). – Charles Duffy Aug 16 '22 at 15:50
  • @meijin3, ...I added an additional duplicate to the list. As it tells you, you can use `u_excl=( --exclude={foo,bar,baz} )`, performing the brace expansion _when creating the array_. – Charles Duffy Aug 16 '22 at 15:54
  • If I'm not replying back, it's only because I'm absorbing the feedback. – meijin3 Aug 16 '22 at 15:57
  • @CharlesDuffy: corrected: https://ideone.com/bcbdKt. (`//#/` => `/#/`) – rici Aug 16 '22 at 16:10
  • I need to research this syntax: `"${u_excl[@]/#/--exclude=}"`. I've never seen it before. It strangely single quotes everything except for the last argument `--exclude=/lost+found `. – meijin3 Aug 16 '22 at 16:28
  • The single quotes will be used when `set -x` prints anything where _not_ including them could make things that are literal data look like shell syntax. In your examples, that's the `*`s in your `/path/*` patterns. But to be clear, that's an artifact of how `set -x` prints things; it's not a change to the contents of your data. – Charles Duffy Aug 16 '22 at 16:29
  • Remember, `set -x`'s goal is to represent an argument list in such a way that you can copy-and-paste it back into the shell and have it parse to the same value as what's actually being run. You don't want `/proc/*` to be changed to `/proc/1 /proc/100 /proc/101` etc when it gets sent back through the parser. – Charles Duffy Aug 16 '22 at 16:32
  • As for the syntax -- see https://wiki.bash-hackers.org/syntax/pe – Charles Duffy Aug 16 '22 at 16:34
  • (a lot of the discussion around `set -x` "adding quotes" reminds me of Python-tag discussion of people being surprised when a string they created to contain `r'\'` prints as `'\\'`; no, an extra backslash wasn't added, it's just a different way of printing the same string -- it's the same thing in bash with `set -x` choosing representations that use single quotes -- the quotes weren't added to your value, they're just used to _print_ the value using one of the language's several available ways to escape literal strings for use as syntax). – Charles Duffy Aug 16 '22 at 16:40
  • In hindsight, the quoting part is very obvious. I was just thrown off by some of the other info there. Taking a look at your link now. – meijin3 Aug 16 '22 at 16:41
  • I've finally had a chance to finish going through the information linked above and understand things much better. Thank you so much to rici and Charles Duffy for all of your assistance! – meijin3 Aug 17 '22 at 12:35

0 Answers0