1

I have a couple of python scripts that are responsible for managing some live feed processing. Its structured like so:

Script 1: manages an "aggregate" list of live events that provides some very thin data about all events.

Script 2: manages a list of threads that process detailed feeds for each live event.

Script 1 is responsible for defining which events are active and (for now) writes all the unique identifiers for the active events to a flat file (not liking that at all). Script 2 reads those unique identifiers, checks to see if it already has a thread with that ID and if not, it starts that thread which then processes the detailed data for that event. Script 2 does NOT define when that thread should be marked as inactive or remove it from the quasi-queue file. The threads know when they should terminate themselves and script 1 monitors a feed that is the master list which defines what events are active. This works but fairly well but it feels clunky and poor to me.

I've looked at this Threading pool similar to the multiprocessing Pool? and Queue approaches like this https://www.ibm.com/developerworks/aix/library/au-threadingpython/ but they don't seem to apply well because the live event threads don't have a specified lifespan...they are spawned and live until their event is over (in the order of hours).

I'm still new to python and this feels a bit over my head. Any kind of sanity/stupidity check you could offer in terms of implementation approaches would be greatly appreciated.

EDIT: I'm not in a position to use an external module because of sys admin limitations :(

Community
  • 1
  • 1
Brad
  • 6,106
  • 4
  • 31
  • 43
  • You can always put a copy of a third-party module into your project. I assume you're allowed to install *your* project on the target system. ;) – Jean-Paul Calderone Oct 18 '13 at 18:50

1 Answers1

1

It sounds like you need to use something like celery.

Marcin
  • 48,559
  • 18
  • 128
  • 201
  • I'm not in a position to use an external module because of sys admin limitations :( – Brad Oct 18 '13 at 16:38
  • 1
    @Brad That's completely untrue unless you cannot write any files and cannot access the shell. – Marcin Oct 18 '13 at 17:53