0

I am working on an Automation POC wherein I have to perform below steps in sequence: 1) create a debian pkg with the help of a tool, and push that to a cloud repository

2) On my prod server I will be executing a python program through a cron job, that will keep monitoring the cloud location and if there is any new file available, it will pull that file from there and install it(debian) on the server from where the python program is running. However, there are possibilities that the cloud repo might not get new file for days. So, in that case even if the repo is updated say 3 days back, but for the python program that is monitoring the cloud location, the file that got uploaded on the cloud will be a latest until a new file comes. So, as a work around I am trying to work on the logic that will keep comparing the timestamp, i.e. if the timestamp of a debian file doesn't change then the python program is suppose to exit/pass , or else execute the business logic.

I have written two programs that satisfies the point (1),and partially satisfies point (2). So going forward, I will be focusing on the point(2) code here. Below is the code that I think should be working at the background to pull the latest file, and if the timestamp is same then it should exit, to test the scenario, I have just taken my machine's local paths:

import os
import subprocess
import glob
latest_file = 0 # initialized latest_file with zero to compare later
new_path = '/home/amitesh/Desktop'
file_path = glob.iglob('/home/amitesh/Desktop/linux_triad/*.deb')

latest_file = max(file_path, key=os.path.getctime) # Now the latest_file variable has a file in it
time_stamp = os.path.getmtime(latest_file)# gives the timestamp of the latest file

a = 0 # initialized it with zero to compare it with time stamp as follows.
while True:
    if a == time_stamp:
        pass
    else:
        subprocess.Popen(['cp', '-r', latest_file, new_path])
        break

In the above code I am just trying to compare two variables 'a' and 'timestamp', i.e. if the value of a is same as time_stamp, then do nothing, and simply pass. else perform the business logic. While I execute the code I did see that a file is copied to the desired location, however, if I check the file timestamp, that doesn;t seems to be the latest one. below are the bunch of files available within the file_path variable.

baqus_0.1-2_amd64.deb Tue 14 May 2019 01:24:02 PM IST
baqus_0.3-1_amd64.deb Tue 14 May 2019 01:24:04 PM IST
baqus_0.4-1_amd64.deb Tue 14 May 2019 01:24:09 PM IST
leesofd_0.1-1_amd64.deb Tue 14 May 2019 01:24:16 PM IST
syslmd_0.3-2_amd64.deb Tue 14 May 2019 01:24:21 PM IST

My code picked up the second file from the list, and it doesn't seems to have the latest timestamp if we see closely. The latest time stamp belongs to syslmd_0.3-2_amd64.deb which is 01:24:21 PM IST. So, again, my code is picking baqus_0.3-1_amd64.deb which has the time stamp 01:24:04 PM IST which lower. I have executed it multiple times just to make sure my observations are correct, and so it is. each time same file is copied to the target location.

So, either my logic is wrong, or I am using wrong functions may be. Kindly suggest.

The changed code inside the while loop

if a == time_stamp:
    pass
else:
    print('copying of the ', latest_file, 'started')
    sleep(4)
    subprocess.Popen(['cp', '-r', latest_file, new_path])
    sleep(3)
    os.system('sudo dpkg --install ' +latest_file)
    a = time_stamp
user3521180
  • 1,044
  • 2
  • 20
  • 45

1 Answers1

2

I'm not sure that I fully understand the reason for needing that workaround - but one thing catches my eye:

You are sorting the files with key=os.path.getctime. Following the docs that is

"on some systems (like Unix) is the time of the last metadata change, and, on others (like Windows), is the creation time for path".

As timestamp from the sorting result, you store os.path.getmtime(latest_file), which is, according to the docs

"Return the time of last modification of path."

According to this question, these two time stamps are not necessarily the same. So the ctime could also depend on the time, the file was copied and that depends on how you push the files in step 1). In contrast, mtime is related to the file contents.

So try to use getmtime in both statements. If that doesn't work, as last resort, you could also name your files like 20190516_095200_sth.deb and sort by file name.

mcrot
  • 446
  • 2
  • 12
  • hello mcrot, thank you for your reply, and yes, I used getmtime() at both the location, and it seems to be fixing my latest file, and timestamp issue. Now I am able to get the most latest file. But now I am facing another issue with the same code. I found out that the program is never satisfying the "if" block, I could say that becasue, as soon as I execute the code, and since the existing file there is same 4 days old, so the program is replacing it again and again. So I made some logic change for the flag a. I have added the modified code in the main post. – user3521180 May 17 '19 at 10:19
  • The change is 'a' = time_stamp. So that once subprocess is done with the execution, the value of 'a' is that of time_stamp. However, as you know the moment I remove "break", the program goes into forever loop. And again, the same file gets update again once, but the time stamp that change this time remians forever as well. Not sure If I could convey my message clear to you. let me know if you have more questions. – user3521180 May 17 '19 at 10:30
  • Sorry, I'm still a bit confused and do not understand what you want to achieve. My suggestion to you is, that you create a new question with a very concrete example: with three files in a folder and their names+time stamps, together with the code which is as simple as possible. Use "print" instead of system calls if the system calls ("Popen()", "system()") are not relevant. Then describe the output you get from that and which specific output you expect instead. Then there is a good chance that others or me understand what's going wrong. – mcrot May 17 '19 at 15:26
  • Ok, agreed, since I have accepted your first answer, so we can conclude the thread here. – user3521180 May 17 '19 at 19:02