How to build script bash with SFTP connection to pull files

Question

I'm implementing agent script bash to pull files from the remote server with SFTP service. The script must:

connect SFTP
file listing
cycling on files found
get every file and copy agent side
after that files copied must be deleted

The script is followed:

 #!/bin/bash

SFTP_CONNECTION="sftp -oIdentityFile=/home/account_xxx/.ssh/service_ssh user@host"

DEST_DATA=/tmp/test/data/

# GET list file by ls command ###############

$SFTP_CONNECTION
$LIST_FILES_DATA_OSM1 = $("ls fromvan/test/data/test_1")
echo $LIST_FILES_DATA_OSM1


for file in "${LIST_FILES_DATA_OSM1[@]}"
       do
                $SFTP_CONNECTION get $file $DEST_DATA
                $SFTP_CONNECTION rm $file
       done

I tried the script but it seems that the connection and command execution (ls) are distinct on thread separated. How can I provide command sequential as described above ?

Screenshoot:

Invalid find command

SSH it seem not available

RSYNC result to take the files is the followed:

Thanks

what problem for install it? But with rsync u can do it more faster with only 2 line code... — beliy, Nov 21 '17 at 16:04
https://www.digitalocean.com/community/tutorials/how-to-copy-files-with-rsync-over-ssh — beliy, Nov 21 '17 at 16:05
I can only access by SFTP on the Server. I can't install nothing — Stefano, Nov 21 '17 at 16:12
As there is such a limitation of the commands available maybe you have less restrictions if you connect via SSH? — M. Becerra, Nov 21 '17 at 16:13
sftp work over SSH, also rsync can be run from any point to sync - need only SSH — beliy, Nov 21 '17 at 16:30
@beliy, not always true. It's possible to lock down SSH to only allow the sftp subsystem, and not programs -- such as rsync -- that require shell access. — Charles Duffy, Nov 21 '17 at 16:33
If you want to copy all files in a local directory to a remote one, or the inverse, you'll have a better time of it using `lftp` as your sftp client. — Charles Duffy, Nov 21 '17 at 16:35
https://stackoverflow.com/a/17826579/14122 looks pretty much on-point. — Charles Duffy, Nov 21 '17 at 16:39

M. Becerra · Answer 1 · 2017-11-21T17:42:47.563

1

First of all, I would recommend the following syntax changes:

#!/bin/bash

sftp_connection() { 
    sftp -oIdentityFile=/home/account_xxx/.ssh/service_ssh user@host "$@"; 
}

Dest_Data=/tmp/test/data/

# GET list file by ls command ###############

sftp_connection
List_Files_D_OSM1=$("ls fromvan/test/data/test_1")
echo "$LIST_FILES_DATA_OSM1"


for file in "${LIST_FILES_DATA_OSM1[@]}"
    do
        sftp_connection get "$file" $Dest_Data
        sftp_connection rm "$file"
    done

Quoting $file and $List_Files_D_OSM1 to prevent globbing and word splitting.
Assignments can't start with a $, otherwise bash will try to execute List_Files_D_OSM1 and will complain with a command not found
No white spaces in assignments like List_Files_D_OSM1 = $("ls fromvan/test/data/test_1")

You can use ShellCheck to catch this kind of errors.

Having said that, it is in general not a good idea to use ls in such way.

What you can use instead is something like find. For example:

find . -type d -exec echo '{}' \;

edited Nov 21 '17 at 17:42

answered Nov 21 '17 at 14:34

M. Becerra

639
6
21

@Stefano what do you mean? How does your code look exactly? You could edit it to your question – M. Becerra Nov 21 '17 at 15:38
it mean that sftp protocol can't perform the command because it not available, My code is the same as described. If I perform the script it try the connetion ad show to screen sftp> . it's waiting the command. I update the post with screenshoot – Stefano Nov 21 '17 at 15:47
This is still really unfortunate from a syntax and best-practices perspective. See [BashFAQ #50](http://mywiki.wooledge.org/BashFAQ/050) re: why commands shouldn't be stored in strings, and [Why you shouldn't parse the output of `ls`](http://mywiki.wooledge.org/ParsingLs), and [the pertinent POSIX standards re: variable names](http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html) -- all-caps names are reserved for variables with meaning to POSIX-specified tools, whereas names with at least one lower-case character are guaranteed safe for application use. – Charles Duffy Nov 21 '17 at 16:30
If you want to shorten things, define a function: `sftp_connection() { sftp -oIdentityFile=/home/account_xxx/.ssh/service_ssh user@host "$@"; }`, and then call `sftp_connection rm "$file"` – Charles Duffy Nov 21 '17 at 16:32

Charles Duffy · Answer 2 · 2017-11-21T16:46:22.537

0

Use a different client. lftp supports sftp as a transport, and has a subcommand for mirroring which will do the work of listing the remote directory and iterating over files for you.

Assuming your ~/.ssh/config contains an entry like:

Host myhost
    IdentityFile /home/account_xxx/.ssh/service_ssh

...you can run:

lftp -e 'mirror -R fromvan/test/data/test_1 /tmp/test/data' sftp://user@myhost

edited Nov 21 '17 at 16:46

answered Nov 21 '17 at 16:42

Charles Duffy

280,126
43
390
441

Original answer didn't tell lftp to use the sftp transport. That's a rather critical piece -- be sure you note the edit. – Charles Duffy Nov 21 '17 at 16:46

How to build script bash with SFTP connection to pull files

2 Answers2