0

I'm writing a RunManager in python which should schedule tasks on a remote server (using SSH. each task is a process).

For the scheduling, I use a shared slots file on the remote server which denotes the number of available slots on that server.

Now, each user runs an instance of the RunManager, so there might be parallel accesses to the slots file. Is there a way for me to atomically increase/decrease the slot count (prevent race-condition)?

Searching 'Mutual protect shared file' on Google yields extreremely unrelated results.

Elad Weiss
  • 3,662
  • 3
  • 22
  • 50

1 Answers1

2

This helpful SO question details a few possible ways you can lock a file in Python, but lots of the answers point out the difficulty of reliably doing so (there's a lot that can go wrong).

I would advise using either an in-memory store (like Redis) or a database, which each of your processes could then talk to. Yet another alternative is to have yet another process that controls the reads and writes, and to which the other processes communicate, but doing that is a fair amount of work, and basically amounts to being a database server in the end.

This seems to me like a situation where you should just use the right tool for the job (a database).

Jonah Bishop
  • 12,279
  • 6
  • 49
  • 74