1

I have a Bash (ver 4.4.20(1)) script running on Ubuntu (ver 18.04.6 LTS) that generates an SCP error. Yet, when I run the offending command on the command line, the same line runs fine.

The script is designed to SCP a file from a remote machine and copy it to /tmp on the local machine. One caveat is that the script must be run as root (yes, I know that's bad, this is a proof-of-concept thing), but root can't do passwordless SCP in my enviroment. User me can so passwordless SCP, so when root runs the script, it must "borrow" me's public SSH key.

Here's my script, slightly abridged for SO:

#!/bin/bash

writeCmd() { printf '%q ' "$@"; printf '\n'; }

printf -v date '%(%Y%m%d)T' -1

user=me
host=10.10.10.100
file=myfile
target_dir=/path/to/dir/$date

# print command to screen so I can see what is being submitted to OS:
writeCmd su - me -c 'scp -C me@$host:/$target_dir/$file.txt /tmp/.'
su - me -c 'scp -C me@$host:/$target_dir/$file.txt /tmp/.'

Output is:

su - me -c scp-Cme@10.10.10.100://.txt/tmp/.

It looks like the ' ' character are not being printed, but for the moment, I'll assume that is a display thing and not the root of the problem. What's more serious is that I don't see my variables in the actual SCP command.

What gives? Why would the variables be ignored? Does the su part of the command interfere somehow? Thank you.

(NOTE: This post has been reedited from its earlier form, if you wondering why the below comments seem off-topic.)

Pete
  • 1,511
  • 2
  • 26
  • 49
  • 2
    `bash -x yourscript` is a very good place to start. This smells like you've got hidden or nonprintable characters that modify how your script is executed. – Charles Duffy Sep 04 '22 at 17:02
  • 1
    If so, this question is duplicative of [Are shell scripts sensitive to encodings and line endings?](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings). (If this is Ubuntu _in WSL_, or you're using Windows-centric editing tools to create your script, consider the strength of that suspicion doubled). – Charles Duffy Sep 04 '22 at 17:03
  • 2
    Another thing: Change `su - me -c '...'` to `su - me -c 'set -x; ...'` so you get trace-level logging inside the script that runs as `me` as well. – Charles Duffy Sep 04 '22 at 17:04
  • 1
    Beyond that, just say no to using `echo` to write commands unless you _really_ know what you're doing. `writeCmd() { printf '%q ' "$@"; printf '\n'; }` is a safer alternative; `writeCmd su - me -c 'scp -C me@10.10.10.100:/path/to/file.txt /tmp/.'` is guaranteed to write something that, when copied-and-pasted back, is behaviorally identical to `su - me -c 'scp -C me@10.10.10.100:/path/to/file.txt /tmp/.'`; that's not true of the echo you're doing now. – Charles Duffy Sep 04 '22 at 17:07
  • 1
    (I describe the `printf %q` approach as "safer" above, but the other thing it is is _more accurate_; it writes output that includes every character in a visible/readable format, which `echo` won't do: if there's a terminal control character that, say, moves the cursor around in your command, when you use `echo` the cursor will just get moved, and perhaps you never find out that that happened; whereas when you use `printf %q` it gets printed in a format that lets you know that it's there). – Charles Duffy Sep 04 '22 at 17:17
  • @CharlesDuffy Wow, thanks Charles, let me tinker with your suggestions, I'll let you know. You rock! – Pete Sep 04 '22 at 17:23
  • @CharlesDuffy Added an Edit to the Original Post; may I ask you to comment? Thanks – Pete Sep 04 '22 at 17:54
  • 1
    So, I did suggest `writeCmd su - me -c '...'`, not `writeCmd "I should su - me -c '...'"`; that distinction is important. :) – Charles Duffy Sep 04 '22 at 17:55
  • 1
    BTW, I left off the `function` on purpose. See https://wiki.bash-hackers.org/scripting/obsolete; there are relevant entries in both 1st and 3rd tables, check them both. – Charles Duffy Sep 04 '22 at 17:55
  • 1
    anyhow -- the most glaring thing observed is that in `scp -C me@://.txt /tmp/.`, most of the filename is missing. Your original command _exactly as-given in the question_ doesn't reproduce that, so there's presumably something you're simplifying that's at fault. Please make sure you're giving us a [mre] -- code we can run _without changes_ to see the same problem ourselves. – Charles Duffy Sep 04 '22 at 17:56
  • 1
    (As for the extra backslashes -- `printf %q 'foo bar'` can write `foo\ bar`, _or_ it can write `'foo bar'`, or in theory it could write, say, `foo' 'bar`; the only guarantee is that it'll write something bash parses back to the same data as its original argument, not that it'll be the same sequence of characters; so some changes to the syntax are expected, but the semantics should be identical; but what we want to pass as its arguments is the actual command you're running, not a string describing that command to a human) – Charles Duffy Sep 04 '22 at 18:00
  • 1
    (btw, if you're running bash 5.0 or later, `writeCmd() { printf '%s\n' "${*@Q}"; }` is a newer form of writeCmd that'll still provide semantically-identical output, but often in a prettier form). – Charles Duffy Sep 04 '22 at 18:02
  • 1
    (Another note about Stack Overflow norms: When editing a question, try to focus on clarity for people seeing it for the first time, instead of making it easy for people who've seen it earlier to understand what was edited; we have really great edit-history tools, and the overarching goal is to build a knowledgebase useful to people in the future. For that reason, we tend to frown a bit on edit markers or edits that try to preserve original context over making the question as accessible to new readers as possible; except when that edit would make an answer no longer make sense) – Charles Duffy Sep 04 '22 at 18:06
  • 1
    (...but insofar as there aren't any answers here yet, the "stay clear of edits that would make an answer no longer make sense" guideline is one you don't need to worry about right now). – Charles Duffy Sep 04 '22 at 18:07
  • 1
    Another note: POSIX _allows_, but doesn't require, `//` at the front of a path to have special meaning distinct from that of `/`. It's unlikely that you're copying files to/from a system that's making use of that -- it's been maybe 15 years since I've run system administration somewhere using an oddball filesystem where it was meaningful -- but it's still _allowed_ for it to change behavior; something to be aware of when troubleshooting. – Charles Duffy Sep 04 '22 at 18:09
  • @CharlesDuffy Thanks Charles, this is all excellent information. Before SO flags this conversation for having too many comments, I'll just say that yes, my original code used variable substitution to build those pathnames, and I skipped that part for the sake of brevity. (When I post, I try to post as concise code as possible.) Now I'm wondering if the var substitution is my problem? The `+ scp -C me@://.txt /tmp/.` output is what the SCP command would look like without the variables subbed in. I'll have to think about this... – Pete Sep 04 '22 at 18:09
  • @CharlesDuffy Agreed! Give me 120 seconds to switch to a diff laptop, I'll be right with you! Thanks – Pete Sep 04 '22 at 18:10
  • @CharlesDuffy Redited the post, as per your recommendation. Thanks again! – Pete Sep 04 '22 at 19:11
  • 1
    BTW, the lack of spaces in the output you have from writeCmd is still odd. You're sure it was tested exactly as given here? Leaving out the space before the closing quote in `'%q '` would do it. – Charles Duffy Sep 05 '22 at 19:29
  • @CharlesDuffy, you were right, I dropped the space in `'%q '` without realizing it. Good catch! – Pete Sep 06 '22 at 20:18

1 Answers1

1

When you run:

writeCmd su - me -c 'scp -C me@$host:/$target_dir/$file.txt /tmp/.'

you'll see that its output is (something equivalent to -- may change version-to-version):

su - me -c scp\ -C\ me@\$host:/\$target_dir/\$file.txt\ /tmp/.

Importantly, none of the variables have been substituted yet (and they're emitted escaped to show that they won't be substituted until after su runs).

This is important, because only variables that have been exported -- becoming environment variables instead of shell variables -- survive a process boundary, such as that caused by the shell starting the external su command, or the one caused by su starting a new and separate shell interpreter as the target user account. Consequently, the new shell started by su doesn't have access to the variables, so it substitutes them with empty values.

Sometimes, you can solve this by exporting your variables: export host target_dir file, and if su passes the environment through that'll suffice. However, that's a pretty big "if": there are compelling security reasons not to pass arbitrary environment variables across a privilege boundary.

The safer way to do this is to build a correctly-escaped command with the variables already substituted:

#!/usr/bin/env bash
#              ^^^^- needs to be bash, not sh, to work reliably

cmd=( scp -C "me@$host:/$target_dir/$file.txt" /tmp/. )
printf -v cmd_v '%q ' "${cmd[@]}"
su - me -c "$cmd_v"

Using printf %q is protection against shell injection attacks -- ensuring that a target_dir named /tmp/evil/$(rm -rf ~) doesn't delete your home directory.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Yes! Exactly! This solved my problem perfectly, thanks so much. Sometimes its the small things we don't think about (like how calling `su` opens a new shell) that kicks us. – Pete Sep 05 '22 at 14:46