3

I am running a python script that may or may not take few hours to complete.

In the beginning of my python script, I want to check if this python script is already running or not.

If it is already running, I want to exit my current python that I just started.

For example:

python started 1AM and keeps on running until 3AM started another one at 2AM without knowing it is already running. I want my 2AM python to check and exit since it is already running.

How can I write this python?


This is what I tried for locking..

try:
    l = lock.lock("/home/auto.py", timeout=600) # wait at most 10 minutes

except error.LockHeld:
    e = sys.exc_info()[0]
    logging.error("Error: " + str(e) + " at main gatering Stats")
    smtpObj.sendmail(sender, receivers, message + "Error: " + str(e) + " at main gatering stats")
    exit("Fail: " + str(e) + " at main gathering Stats")
else:
    l.release()

so I thought this will wait for 10 minutes if it is still running then exit.. if it is not running anymore, then run the current python

Tim
  • 103
  • 2
  • 9

1 Answers1

2

You can try using the lockfile-create command with the r flag to retry a specified amount of times catching a CalledProcessError and exiting, the -p flag will store the pid of the process :

import os
import sys
from time import sleep

from subprocess import check_call, CalledProcessError

try:
    check_call(["lockfile-create", "-q","-p", "-r", "0", "-l", "my.lock"])
except CalledProcessError as e:
    print("{} is already running".format(sys.argv[0]))
    print(e.returncode)
    exit(1)


# main body

for i in range(10):
    sleep(2)
    print(1)

check_call(["rm","-f","my.lock"])

Running a test.py script with the code above while one is already running outputs the following:

$ python  lock.py 
lock.py is already running
4

Options

-q, --quiet

Suppress any output. Success or failure will only be indicated by the exit status.

-v, --verbose

Enable diagnostic output.

-l, --lock-name

Do not append .lock to the filename. This option applies to lockfile-create, lockfile-remove, lockfile-touch, or lockfile-check.

-p, --use-pid

Write the current process id (PID) to the lockfile whenever a lockfile is created, and use that pid when checking a lock's validity. See the lockfile_create(3) manpage for more information. This option applies to lockfile-create, lockfile-remove, lockfile-touch, and lockfile-check.

-o, --oneshot

Touch the lock and exit immediately. This option applies to lockfile-touch and mail-touchlock. When not provided, these commands will run forever, touching the lock once every minute until killed.

-r retry-count, --retry retry-count

Try to lock filename retry-count times before giving up. Each attempt will be delayed a bit longer than the last (in 5 second increments) until reaching a maximum delay of one minute between retries. If retry-count is unspecified, the default is 9 which will give up after 180 seconds (3 minutes) if all 9 lock attempts fail.

Description

The lockfile_create function creates a lockfile in an NFS safe way.

If flags is set to L_PID then lockfile_create will not only check for an existing lockfile, but it will read the contents as well to see if it contains a process id in ASCII. If so, the lockfile is only valid if that process still exists.

If the lockfile is on a shared filesystem, it might have been created by a process on a remote host. Thus the process-id checking is useless and the L_PID flag should not be set. In this case, there is no good way to see if a lockfile is stale. Therefore if the lockfile is older then 5 minutes, it will be removed. That is why the lockfile_touch function is provided: while holding the lock, it needs to be refreshed regularly (every minute or so) by calling lockfile_touch ().

The lockfile_check function checks if a valid lockfile is already present without trying to create a new lockfile.

Finally the lockfile_remove function removes the lockfile.

The Algorithm

The algorithm that is used to create a lockfile in an atomic way, even over NFS, is as follows:

1

A unique file is created. In printf format, the name of the file is .lk%05d%x%s. The first argument (%05d) is the current process id. The second argument (%x) consists of the 4 minor bits of the value returned by time(2). The last argument is the system hostname.

2

Then the lockfile is created using link(2). The return value of link is ignored.

3

Now the lockfile is stat()ed. If the stat fails, we go to step 6.

4

The stat value of the lockfile is compared with that of the temporary file. If they are the same, we have the lock. The temporary file is deleted and a value of 0 (success) is returned to the caller.

5

A check is made to see if the existing lockfile is a valid one. If it isn't valid, the stale lockfile is deleted.

6

Before retrying, we sleep for n seconds. n is initially 5 seconds, but after every retry 5 extra seconds is added up to a maximum of 60 seconds (an incremental backoff). Then we go to step 2 up to retries times.

There seems to be an equivalent package called lockfile-progs on redhat.

On mac you could use lockfile and do something like:

import os
import sys
from time import sleep
import os
from subprocess import Popen, CalledProcessError, check_call


p = Popen(["lockfile", "-r", "0", "my.lock"])
p.wait()
if p.returncode == 0:
    with open("my.pid", "w") as f:
        f.write(str(os.getpid()))
else:
    try:
        with open("my.pid") as f:
            # see if process is still running or lockfile
            # is left over from previous run.
            r = f.read()
            check_call(["kill", "-0", "{}".format(r)])
    except CalledProcessError:
        # remove old lock file and create new
        check_call(["rm", "-f", "my.lock"])
        check_call(["lockfile", "-r", "0", "my.lock"])
        # update pid
        with open("my.pid", "w") as out:
            out.write(str(os.getpid()))
        print("Deleted stale lockfile.")
    else:
        print("{} is already running".format(sys.argv[0]))
        print(p.returncode)
        exit(1)
# main body

for i in range(10):
    sleep(1)
    print(1)
check_call(["rm", "-f", "my.lock"])

In your case maybe using a socket would work:

from socket import socket, gethostname, error, SO_REUSEADDR, SOL_SOCKET
from sys import argv
import  errno



sock = socket()

# Create a socket object
host = gethostname()  
# /proc/sys/net/ipv4/ip_local_port_range is  32768  61000 on my Ubuntu Machine
port = 60001  
# allow connection in TIME_WAIT
sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)

try:
    sock.bind((host, port))
    sock.connect((host, port))
except error as e:
    # [Errno 99] Cannot assign requested address
    if e.errno == errno.EADDRNOTAVAIL:
        print("{} is already running".format(argv[0]))
        exit(1)
    # else raise the error
    else:
        raise e

# main body
from time import sleep

while True:
    print(1)
    sleep(2)

sock.close()
Community
  • 1
  • 1
Padraic Cunningham
  • 176,452
  • 29
  • 245
  • 321
  • so this will run for the first python and on the scond python it will exit out? also at `heck_call(["lockfile-create","-r", "3", "--lock-name","my.lock"])` what is `--lock-name` and `my.lock`? is this just any name I can create? – Tim Apr 08 '15 at 18:42
  • @Tim, the name can be anything, it is created when the script is run. Try it with the example I provided – Padraic Cunningham Apr 08 '15 at 18:44
  • Traceback (most recent call last): File "./automate.py", line 430, in check_call(["lockfile-create","-r", "3", "--lock-name","my.lock"]) File "/usr/lib64/python2.7/subprocess.py", line 537, in check_call retcode = call(*popenargs, **kwargs) File "/usr/lib64/python2.7/subprocess.py", line 524, in call return Popen(*popenargs, **kwargs).wait() File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1308, in _execute_child raise child_exception` – Tim Apr 08 '15 at 18:47
  • Can you run `lockfile-create` from bash? – Padraic Cunningham Apr 08 '15 at 18:54
  • you mean just `lockfile-create` ? no `-bash: lockfile-create: command not found` – Tim Apr 08 '15 at 18:59
  • what are you running this on? – Padraic Cunningham Apr 08 '15 at 19:01
  • I am practicing it on my macbook which is unix.. but the school server is linux ubuntu – Tim Apr 08 '15 at 19:03
  • I don't use mac so cannot comment on that but I am using Ubuntu and it works as expected. What are you going to be running it on? – Padraic Cunningham Apr 08 '15 at 19:05
  • do you have lockfile installed on mac? – Padraic Cunningham Apr 08 '15 at 19:05
  • I realized I don't have it installed and by using ssh to the school server, it wont let me install it either.. so I guess I have to look for other way of doing it – Tim Apr 08 '15 at 19:11
  • You don't have to install it on ubuntu, it should already be there. You can install lockfile on mac as in the second part of the answer – Padraic Cunningham Apr 08 '15 at 19:12
  • `sudo apt-get install procmail` wont do it either same as yum.. just realized our school sever dont have lockfile function and cant install it – Tim Apr 08 '15 at 19:17
  • I don't get what you mean by won't do it either? What happens? – Padraic Cunningham Apr 08 '15 at 19:32
  • can't install procmail to the school server.. and it is not already installed – Tim Apr 08 '15 at 19:38
  • I thought the school used an Ubuntu server? – Padraic Cunningham Apr 08 '15 at 19:39
  • i think they are.. but it is not installed in there – Tim Apr 08 '15 at 19:50
  • Linux version 3.10.0-123.20.1.el7.x86_64 – Tim Apr 08 '15 at 20:05
  • does `cat /etc/redhat-release` output anything? – Padraic Cunningham Apr 08 '15 at 20:06
  • Red Hat Enterprise Linux Server release 7.0 (Maipo) – Tim Apr 08 '15 at 20:12
  • then you are using redhat not ubuntu. There are a few issues in the script like if it crashes the lock file won't be deleted so I will iron those out later or tomorrow. for now install http://man.flashnux.com/en/redhat/9/9.0/man1/lockfile.1.html – Padraic Cunningham Apr 08 '15 at 20:16