0

I am trying to download a bunch of files from a sftp server. The organization of the server is the following: there is a folder per year, and each yearly folder, there is a folder per day. In each day, there are 8 files ending in .nc that I want to download. The names of the files themselves are too crazy to keep track of. I have tried a few different approaches but I have not been successful in giving get the right instructions to get the files. My current version is to have a loop before connecting to the sftp so I can write the name and then connect to the sftp and download that file:

for i in `seq 1 1 366`;
        do
            if [ $i -lt 10 ]; then
                today='00$i/*.nc' 
            elif [ $i -ge 10 ] && [ $i -lt 100 ]; then
                today= '0$i/*.nc' 
            elif [ $i -ge 100 ]; then
                today= '$i/*.nc' 
            fi
        done

sshpass -p $(cat ~/Code/sftp_passwd.txt) sftp cgerlein@cygnss-sftp-1.engin.umich.edu <<EOF

cd ..
cd ..
cd /data/cygnss/products/CDR/l1/2019/

get $today /scratch/myfolder/ 
quit
EOF

I don't think get is liking the wildcard in there. And this might not even be the best approach. Any suggestions? Thanks!

Cynthia GS
  • 522
  • 4
  • 20
  • Variables aren't expanded inside single quotes, only double quotes. – Barmar Oct 11 '19 at 15:54
  • BTW, you can use `$(printf '%03d' $i)` to get a number with leading zeroes, instead of the `if` statements. – Barmar Oct 11 '19 at 15:55
  • I think you need to use the `mget` command to get multiple files with a wildcard. – Barmar Oct 11 '19 at 15:55
  • @Barmar Thanks, the double quotes + `mget` works but now it downloads all the files from the first folder and then it stops. I tried putting the `sshpass` [ ...] `quit EOF` inside teh `do` loop but that doesn't work. – Cynthia GS Oct 11 '19 at 16:02
  • Ah, I think it was just something weird with the tabulations, but it now seems to work. Thanks! – Cynthia GS Oct 11 '19 at 16:05
  • You can't indent the `EOF` token in a here-doc unless use use `<<-EOF`. – Barmar Oct 11 '19 at 16:05

1 Answers1

3

You can use printf() to format the filename with leading zeroes.

The sftp command needs to be inside the loop.

You can't have a space after today=.

There's no need for cd .. before changing to an absolute pathname.

for i in {1..366}
do
    today=$(printf "%03d/*.nc" $i)
    sshpass -p $(cat ~/Code/sftp_passwd.txt) sftp cgerlein@cygnss-sftp-1.engin.umich.edu <<EOF
cd /data/cygnss/products/CDR/l1/2019/
get $today /scratch/myfolder/ 
quit
EOF
done

BTW, you can do wildcard downloads with curl, see downloading all the files in a directory with cURL

Barmar
  • 741,623
  • 53
  • 500
  • 612