Linux shell script for delete old files from ftp

Question

There is a problem - need to store the database backup on the FTP. On the FTP should not be more than 10 back-ups, ie, After you add backup to FTP should be removed, the oldest files to make the total number of files can not exceed 10 pieces. How can we implement such a removal from the ftp? I'm trying to write a script, but does not work delete:

x=1
ftp -vn $FTP_SERVER<<!
user $FTP_LOGIN $FTP_PASSWORD
binary
put $DUMP_FILE_NAME

for i in `ls -t` do
    if [ $x -le $keep ] then
        ((x++))
        continue
    fi
    delete $i
done

bye
EOF
</i>

Answer 1

This is a script I wrote to remove any files on a remote ftp site older than 7 days. It works by retrieving a listing of the directory, parsing the modified date, and then re-connecting to delete any files older than ndays.

I suspect that the numbers hard-coded into the loop (element date) may change depending on the setup of your system. The return formatting of the ls command is dependent on the local system settings.

Assuming your backups are every day, then setting ndays to 10 might solve your problem.

#!/bin/bash
# get a list of files and dates from ftp and remove files older than ndays
ftpsite="ftp.yourserver.com"
ftpuser="loginusername"
ftppass="password"
putdir="/public_ftp/admin/logs"

ndays=7


# work out our cutoff date
MM=`date --date="$ndays days ago" +%b`
DD=`date --date="$ndays days ago" +%d`


echo removing files older than $MM $DD

# get directory listing from remote source
listing=`ftp -i -n $ftpsite <<EOMYF 
user $ftpuser $ftppass
binary
cd $putdir
ls
quit
EOMYF
`
lista=( $listing )

# loop over our files
for ((FNO=0; FNO<${#lista[@]}; FNO+=9));do
  # month (element 5), day (element 6) and filename (element 8)
  #echo Date ${lista[`expr $FNO+5`]} ${lista[`expr $FNO+6`]}          File: ${lista[`expr $FNO+8`]}

  # check the date stamp
  if [ ${lista[`expr $FNO+5`]}=$MM ];
  then
    if [[ ${lista[`expr $FNO+6`]} -lt $DD ]];
    then
      # Remove this file
      echo "Removing ${lista[`expr $FNO+8`]}"
      ftp -i -n $ftpsite <<EOMYF2 
      user $ftpuser $ftppass
      binary
      cd $putdir
      delete ${lista[`expr $FNO+8`]}
      quit
EOMYF2


    fi
  fi
done

Answer 2

This one is dealing with sftp and has proper date check as the script from @Graeme is only working within one month:

#!/bin/bash
# get a list of files and dates from ftp and remove files older than ndays
ftpsite="sftp -b-  -oIdentityFile=<KEYFILE> -oPort=<PORT>  <USER>@<HOST>"
putdir="/data"

ndays=19

# work out our cutoff date
MM=`date --date="$ndays days ago" +%b`
DD=`date --date="$ndays days ago" +%d`
TT=`date --date="$ndays days ago" +%s`

echo removing files older than $MM $DD

# get directory listing from remote source
echo "
cd $putdir
ls -l
"|$ftpsite >dirlist

# skip first three and last line, ftp command echo
listing="`tail -n+4 dirlist|head -n-1`"

lista=( $listing )

# loop over our files
for ((FNO=0; FNO<${#lista[@]}; FNO+=9));do
  # month (element 5), day (element 6) and filename (element 8)
  # echo Date ${lista[`expr $FNO+5`]} ${lista[`expr $FNO+6`]}          File: ${lista[`expr $FNO+8`]}

  fdate="${lista[`expr $FNO+5`]} ${lista[`expr $FNO+6`]} ${lista[`expr $FNO+7`]}"
  sdate=`date --date="$fdate" +%s`
  # check the date stamp
  if [ $sdate -lt $TT ]
  then
      # Remove this file
      echo "$MM $DD: Removing  ${lista[`expr $FNO+5`]} /  ${lista[`expr $FNO+6`]} / ${lista[`expr $FNO+8`]}"
      $ftpsite <<EOMYF2
      cd $putdir
      delete ${lista[`expr $FNO+8`]}
      quit
EOMYF2

  fi
done

Answer 3

It has passed several years, and while Graeme's answer helped me but I faced some problems with it and here is how to fix them.

Sometimes the sentence like Trying aaaa:bbb:ccc:dddd::0... is at first line of listing varable. You have to remove it.

rem="Trying aaaa:bbb:ccc:dddd::0...";
rep="";
listing=${listing/$rem/$rep}

Also jsan commented he faced other issue: "value too great for base (error token is "08")" when $DD is 0x it is interpreted as x in octal (problem for 8 and 9)

I faced same problem. So you better make sure there is no nun-numeric characters in $MM or $DD and then remove the 0 from beggining. Like 08 should become 8.

so before looping on files, add this code:

MM=$(tr -dc '0-9' <<< $MM)
DD=$(tr -dc '0-9' <<< $DD)
MM=${MM#0}
DD=${DD#0}

First two lines remove non-numeric characters and second last lines fix the problem with 08

Answer 4

Here's a shorter solution using lftp, that'll also work with subdirectories:

#!/bin/bash
# get a list of files and dates from ftp and remove files older than ndays
ftpsite="ftpback-xxx.ovh.net"
ftpuser="user"
ftppass="pass"
#remote folder in which you want to delete files
putdir="/"
nullfolder="/tmp/null"

ndays=19

mkdir -p nullfolder
MM=`date --date="$ndays days ago" +%b`
DD=`date --date="$ndays days ago" +%d`

echo removing files older than $MM $DD
#The no-ssl is there because it's on a local network and remote doesn't support ftp
lftp -e "set ftp:ssl-allow no; mirror $putdir $nullfolder --older-than=now-${ndays}days --Remove-source-files; exit" -u $ftpuser,$ftppass $ftpsite

rm $nullfolder/* -Rf

It has two drawbacks though:

• It will first download those old files (before removing them later), so it uses some more bandwidth
• Those old files will take some space on the local disk before being removed. You can use nullfs to pally to that

Answer 5

Don't use FTP (for many reasons). However, if you do that in ssh, you get a tty, environment, and everything else. Also, if you want a file 10 days old to be deleted (and not one 10 calendar days stale) a simple find -mtime +10 -delete would replace that for loop. So:

ssh user@host 'find $DIR -mtime +10 -delete'

If you want FTP, look at NcFTP, if I recall correctly there were some pretty decent scripting capabilities there.