1

I have a complex command I am passing via ssh to a remote server. I am trying to unzip a file and then change its naming structure and extension in a second ssh command. The command I have is:

ssh root@server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print $1$3".log"}'"

Obviously the " around the .log portion of the print statement are failing me. The idea is that I would strip the .out portion from the filename and end up with file20171119.log as an ending result. I am just a bit confused on the syntax or on how to escape that properly so bash interprets the .log appropriately.

Chris Moretti
  • 585
  • 3
  • 13
  • 31
  • Do you even need to expand the archive? If all you are doing is working with the file name, `bash` parameter expansion can handle that. If `f=/tmp/file.out-20171119.gz`, then `g=${f/.out-/}; g=${g#/tmp/}; g=${g/%.gz/.log}` should do it. – chepner Nov 29 '17 at 17:49
  • There is a .out file in the archive. That is the file I am trying to rename basically. Would that still apply? – Chris Moretti Nov 29 '17 at 17:51
  • Same approach, yes. Different value of `f`, of course, and you may need to adjust the exact patterns I used the the assignments to `g`. – chepner Nov 29 '17 at 17:52
  • Is there a reason you're passing this on the command line rather than stdin? – Charles Duffy Nov 29 '17 at 17:54
  • BTW, are you actually wanting to *`cat`* that file into `awk`? Because that's not what your script does at all. `echo file*` is echoing the **names** of files into `awk`. – Charles Duffy Nov 29 '17 at 17:55
  • @CharlesDuffy He's renaming a file, not reading it. – chepner Nov 29 '17 at 17:57
  • Ahh. Then why isn't there a `mv` anywhere? – Charles Duffy Nov 29 '17 at 17:58
  • I was originally playing with mv, but could not figure out a syntax that would allow me to use a wildcard. If I wanted to rename the files of anything that started with file to have a new extension...I would want to use `mv file*.out file*.log` It becomes even more complicated when I am trying to remove a section of the filename from the center. – Chris Moretti Nov 29 '17 at 18:04
  • @user1943674, if that's what you want to do, see [BashFAQ #30](http://mywiki.wooledge.org/BashFAQ/030). – Charles Duffy Nov 29 '17 at 18:06
  • @user1943674, ...your current `awk` code generates a bunch of new names (potentially, if actually run in a directory that has files matching the glob), but it just prints them on stdout; it doesn't actually rename any files to use those updated names. – Charles Duffy Nov 29 '17 at 18:08
  • That is a really good point. I was testing this and only saw stdout results and thought it was what I was looking for....definitely not the case. So I need to translate that syntax to mv in some way – Chris Moretti Nov 29 '17 at 18:14
  • *Do* see the above-linked FAQ, if you haven't already. Generating shell commands with `awk` is generally buggy, often in security-impacting ways; it's not a good road to go down. – Charles Duffy Nov 29 '17 at 18:15

3 Answers3

5

The easiest way to deal with this problem is to avoid it. Don't bother trying to escape your script to go on a command line: Pass it on stdin instead.

ssh root@server1 bash -s <<'EOF'
  gzip -d /tmp/file.out-20171119.gz
  # note that (particularly w/o a cd /tmp) this doesn't do anything at all related to the
  # line above; thus, probably buggy as given in the original question.
  echo file* | awk -F'[.-]' '{print $1$3".log"}'
EOF

A quoted heredoc -- one with <<'EOF' or <<\EOF instead of <<EOF -- is passed literally, without any shell expansions; thus, $1 or $3 will not be replaced by the calling shell as they would with an unquoted heredoc.

If you don't want to go the avoidance route, you can have the shell do the quoting for you itself. For example:

external_function() {
  gzip -d /tmp/file.out-20171119.gz
  echo file* | awk -F'[.-]' '{print $1$3".log"}'
}

ssh root@server1 "$(declare -f external_function); external_function"

declare -f prints a definition of a function. Putting that function literally into your SSH command ensures that it's run remotely.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • This seems ideal, but for some reason bash is complaining about that EOF. It says wanted `EOF' followed by an unexpected end of file. – Chris Moretti Nov 29 '17 at 18:06
  • I'd need to see the code *exactly* as it's run to diagnose that. If the final `EOF` (terminating the heredoc) is indented, for example, there's your problem; it has to be at the very beginning of the line that it's on. – Charles Duffy Nov 29 '17 at 18:06
  • @user1943674, ...which is to say, please verify that you get a bug with my code **exactly as it's given here**, without making any changes whatsoever (except for the name of `server1`). – Charles Duffy Nov 29 '17 at 18:08
  • Definitely no bug. It was a spacing issue. Is it possible to also pass in a variable into the shell command? I would like to have the gzip command be run as `gzip -d /tmp/$file` – Chris Moretti Nov 29 '17 at 18:11
  • Yes, it is, and we have questions elsewhere in the knowledgebase asking and answering that question; let me find one... – Charles Duffy Nov 29 '17 at 18:12
  • @user1943674, see [Passing a variable to a remote host in a bash script with ssh and EOF](https://stackoverflow.com/questions/37103664/passing-a-variable-to-a-remote-host-in-a-bash-script-with-ssh-and-eof). Short form: `printf -v file_q '%q' "$file"`, then `ssh user@host "bash -s $file_q" <<'EOF'` and you can refer to the filename as `$1` within the script. – Charles Duffy Nov 29 '17 at 18:13
1

You need to escape the " to prevent them from closing your quoted string early, and you need to escape the $ in the awk script to prevent local parameter expansion.

ssh root@server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print \$1\$3\".log\"}'"
chepner
  • 497,756
  • 71
  • 530
  • 681
  • I get an error now: bash -c unexpected EOF while looking for matching `'' and another error: syntax error: unexpected end of file – Chris Moretti Nov 29 '17 at 17:52
  • Double-check your quoting; works for me (at least, when I replace `gzip` with a variable assignment and echo that value instead of `file*` to `awk`). – chepner Nov 29 '17 at 17:57
  • probably you left an open quote, that made to skip over the proper `EOF` string. – Luis Colorado Nov 30 '17 at 13:18
0

The most probable reason (as you don't show the contents of the root home directory in the server) is that you are uncompressing the file in the /tmp directory, but feeding to awk filenames that should exist in the root home directory.

" allows escaping sequences with \. so the correct way to do is 

ssh root@server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print \$1\$3\".log\"}'"

(like you wrote in your question) this means the following command is executed with a shell in the server machine.

gzip -d /tmp/file.out-20171119.gz; echo file* | awk - F'[.-]' '{print $1$3".log"}'

You are executing two commands, the first to gunzip /tmp/file.out-2017119.gz (beware, as it will be gunzipped in /tmp). And the second can be the source for the problem. It is echoing all the files in the local directory (this is, the root user home directory, probably /root in the server) that begin with file in the name (probably none), and feeding that to the next awk command.

As a general rule.... test your command locally, and when it works locally, just escape all special characters that will go unescaped, after being parsed by the first shell.

another way to solve the problem is to use gzip(1) as a filter... so you can decide the name of the output file

ssh root@server1 "gzip -d </tmp/file.out-20171119.gz >file20171119.log"

this way you save an awk(1) execution just to format the output file. Or if you have the date from an environment variable.

DATE=`date +%Y%m%d`
ssh root@server1 "gzip -d </tmp/file.out-${DATE}.gz >file${DATE}.log"

Finally, let me give some advice: Don't use /tmp to uncompress files. /tmp is used by several distributions as a high speed temporary dir. It is normally ram based, too quick, but limited space, so uncompressing a log file there can fill up the memory of the kernel used for the ram based filesystem, which is not a good idea. Also, a log file normally expands a lot and /tmp is a local system general directory, where other users can store files named file<something> and you can clash with those files (in case you do searches with wildcard patterns, like you do in your command) Also, it is common once you know the name of the file to assign it to environment variables and use those variables, so case you need to change the format of the filename, you do it in only one place.

Luis Colorado
  • 10,974
  • 1
  • 16
  • 31