2

When I spin up a new server I use a cloud-init script to automate the process. I follow progress of this process with:

$ tail -f /var/log/cloud-init-output.log

When it comes to fetching a backup file using scp and piping it through tar -zx I tried the following in the cloud-init script:

$ scp myuser@123.45.6.789:path/to/file.tar.gz /dev/stdout | tar -zx

Whilst this command works, the handy progress indications that scp outputs do not appear in the tail -f output... i.e. I do not see progress like:

file.tar.gz   57%   52MB  25.2MB/s   00:02

I have also tried bash process substitution like so:

$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx)

And still the progress indications do not appear in the tail -f output.

How do I preserve the progress indications in the tail -f output? Particularly to see progress when fetching the larger backup files, it would be really handy to preserve these indications.

Note that when I run both of the above in a bash script directly (with set -x at the top), progress does show for the 'bash process substitution' variant but not the 'piping' variant... but when tailing the cloud-init logs, progress shows for neither variant.

It looks like cloud-init just sends both stdout and stderr from all cloud-init stages to /var/log/cloud-init-output.log (see here).

So we can re-create the cloud-init process (for the purposes of this question) with the following call:

$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx) >output.log 2>&1

and we then follow this log file separately with:

$ tail -f output.log

Strange thing is... no progress status ever appears in this output.log file. And yet progress status is definitely being sent... the only change in the following is to direct to /dev/stdout rather than output.log:

$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx) >/dev/stdout 2>&1
file.tar.gz   57%   52MB  25.2MB/s   00:02

So why do we see progress status on /dev/stdout but not in output.log when we direct stdout to output.log?

drmrbrewer
  • 11,491
  • 21
  • 85
  • 181
  • 1
    `scp` turns off the progress meter if stdout isn't a TTY (which your file is not). It looks like you could convince it that the output is a TTY by using `script` (or something else to run the program under a PTY), see https://stackoverflow.com/a/1402389 – Hasturkun Dec 08 '21 at 16:40
  • 1
    Genius @Hasturkun! I didn't know that a command could change it's output like that, without an explicit option! FYI the helper function in [this answer](https://stackoverflow.com/a/20401674/4070848) worked a treat for me, adding the `-e` option as suggested in a comment... so then I can just prepend the normal command with the helper: `fake_tty scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx) >output.log 2>&1` and the progress is now visible in `output.log`! (I assume it will work the same in actual cloud-init but I haven't tried yet). If you post an answer, I'll accept it... thanks! – drmrbrewer Dec 08 '21 at 17:18
  • OK revise that... it works *without* the `-e` option... i.e. just `script -qfc` (exactly as per the [linked answer](https://stackoverflow.com/a/20401674/4070848))... for some reason using `script -qfce` gives a `No such file or directory error`. – drmrbrewer Dec 08 '21 at 19:52
  • 1
    You probably want to use `-qfec`, since `-c` wants an argument (so I think is taking `e` as one otherwise). I think it'd be better if I mark this as a duplicate of the other (since my only contribution here is the fact that `scp` won't do progress to a non TTY), rather than making an link-mostly answer. – Hasturkun Dec 08 '21 at 20:49
  • Ah, yes you're right about the order qfec... that works. I'm not sure this is really a duplicate... it's a new teaching that scp disables the progress meter to non TTY, even if (once you know that) the answer is the same as the other one. I.e. different question, same answer. – drmrbrewer Dec 08 '21 at 21:22

1 Answers1

1

Like some other tools, scp checks if its output is going to a TTY (by use of isatty), disabling the progress meter output if it is not (you can find similar cases e.g., ls --color=auto emits color codes only when output goes to a terminal).

You can trick it into thinking it is outputting to a TTY by running it under script (as shown in this answer), or any other tool (e.g expect) that runs a program with its output connected to a PTY.

Hasturkun
  • 35,395
  • 6
  • 71
  • 104
  • Thanks! For a bonus point, could you explain why a space is required after `%q` in `script -qfc "$(printf "%q " "$@")" /dev/null`? – drmrbrewer Dec 09 '21 at 17:40
  • 1
    Without the space arguments would get glued together, as `bash` `printf` reuses the format for all arguments. – Hasturkun Dec 09 '21 at 18:11
  • Ah OK I see what it's doing. You end up with a space at the end of the final argument, but this doesn't matter I suppose. – drmrbrewer Dec 09 '21 at 18:58