Bash: How to parse a log file that contains the output of ls -ltr using bash script to extract the file names that are modified before specific time

Question

I am trying to get all the files in SFTP directories that were last modified before 24 hours using Unix bash script. Please note that lftp option is not supported in my environment. So, I have created a script that will list all the sub directories of the SFTP directory to a local file, then I have used a loop to traverse all those sub directories names, printed the content of the directory using ls -ltr and directed that output to a local file.

Now I have a file with below content and let's assume that the current time is Oct 28th 22:32.

-rw------- 1 200      100             1930 Oct 25 08:31 File1
-rw------- 1 200      100              280 Oct 25 11:32 File2
-rw------- 1 200      100              280 Oct 25 12:17 File3
-rw------- 1 200      100              280 Oct 25 22:31 File4
-rw------- 1 200      100              280 Oct 26 22:32 File5
-rw------- 1 200      100              280 Oct 27 22:31 File6
-rw------- 1 200      100              280 Oct 16 09:59 File7
-rw------- 1 200      100              280 Oct 16 09:59 File8
-rw------- 1 200      100              280 Oct 16 10:00 File9

Is it possible to get all the filenames and their last modified time stamps with last modified date < sysdate - 24 hours?

It would really be better if you could [avoid parsing `ls` output](http://mywiki.wooledge.org/ParsingLs) — tripleee, Oct 28 '19 at 14:39
@tripleee Are you aware of another way to get this information from an sftp connection? This isn't ls(1). This is the "ls" command inside of sftp. — Rob Napier, Oct 28 '19 at 14:40
[https://stackoverflow.com/questions/16085958/find-the-files-that-have-been-changed-in-last-24-hours] refer above and avoid parsing ls and instead use find. — Piyush, Oct 28 '19 at 15:47
I used to suffer under similar constraints in a security obbsessed environment. But they did let me move files to an `archive` area once they had been processed. Maybe you can rebuild to processes to "hide" files once they have been processed, then you only need to keep polling for new files and process as they arrive. Just an idea. Good luck. — shellter, Oct 28 '19 at 22:59

score 0 · Answer 1 · answered Oct 28 '19 at 15:40

For automation, I strongly suggest using 'lftp'. The 'sftp' is a simple interface, appropriate for interactive usage (e.g., no status code for failures, etc.).

The 'lftp' has lot of options beyond 'sftp'. In particular, it has 'client side' listing (cls):

sftp
open sftp://host/path
user user password
cls -l --time-style='%Y-%m-%d %H:%M:%S'
Output:
drwxr-xr-x    2 root     root         4096 2019-10-13 16:01:46 /bin/
drwxr-xr-x    3 root     root         4096 2019-06-30 08:07:02 /boot/
drwxr-xr-x    2 root     root         4096 2019-06-09 19:36:33 /cdrom/
drwxr-xr-x   18 root     root         3960 2019-10-19 08:08:01 /dev/
drwxr-xr-x  150 root     root        12288 2019-10-20 15:49:39 /etc/
drwxr-xr-x    4 root     root         4096 2019-06-09 23:18:04 /home/

If you are looking for cutoff, you can also sort by time (I believe --sort=date, and optioanlly reverse -r), and stop processing once you match the date range.

Side note: sftp has a limited 'find' command. Unfortunately, it does not offer filtering by date.

Most other 'sftp' clients that I've tried were using the default LS format. The exception are the SFTP modules for Perl, Python, which provide the client with the date in "Unix" format, allowing any test. However, this is significantly more work than using lftp.

LFTP is not available, that's the reason I am exploring other options to get this done. — D789rul, Oct 28 '19 at 16:34

score 0 · Accepted Answer · answered Nov 13 '19 at 21:55

I ended up parsing the log file line by line, extracted the last modified time to a variable and file name to another variable. Then I converted the last modified time string to Unix time format and then to epoch number. Similarly I got the epoch number for sysdate-1. Finally I compared those two values and filtered the file names matched my criteria. code snippet is given below

prev_day_time=$(date +%s -d "$(date --date='1 day ago')")
file_mod_time=$(date +%s -d "$(date -d "$(echo $(echo $files_list | awk '{print $1}') | sed -e 's/\(.....\)/\1 /')")")

Bash: How to parse a log file that contains the output of ls -ltr using bash script to extract the file names that are modified before specific time

2 Answers2