1

I have a directory with several python modules in it. Each module is mutually exclusive of all the others but after lots of trial and error, I have decided that each module horks out when using the multiprocessing functionality in Python. I have used the join() function on each process and its just not working like I want.

What I am really looking for is the ability to drop new mutually exclusive python modules in to the directory and have them invoked when the directory is launched. Does anyone know how to do this?

juliomalegria
  • 24,229
  • 14
  • 73
  • 89
aeupinhere
  • 2,883
  • 6
  • 31
  • 39
  • What exactly is the problem; `multiprocessing` troubles or how to detect new files popping up in a directory? – Fred Foo Jan 03 '12 at 21:08
  • I suppose at this point, I need to detect and launch new files in a directory so that the directory itself looks like a single module. I know this sounds crazy which is why I am reaching out to the community. – aeupinhere Jan 03 '12 at 21:14
  • 2
    What is 'horks'? What does it mean to launch a directory? –  Jan 03 '12 at 21:28
  • Wow this is unintelligible. Adam, you should clarify what you want before writing it down. – Tobu Jan 03 '12 at 21:33
  • Horks means that it one of the processes has the ability to break the others. I want to develop a plug and play architecture that will allow new modules to be detected and run when they are placed in the appropriate directory. And I do apologize for being unintelligible. It's flu season and I am not 100% ;-) – aeupinhere Jan 03 '12 at 21:39

1 Answers1

2

It sounds to me like you are asking about plugin architecture and sandboxing. Does that sound right?

The plugin component has been done and written about else where. SO has code examples on basic ways to import all the files.

The sandbox part is going to be a harder. Have a look at RestrictedPython and the Restricted Execution docs and the generally older but nevertheless helpful discussion of sandboxing.

If you aren't worried about untrusted code but rather want to isolate errors you could just wrap each module in a generic try/except that handles all errors. This would make debugging hard but would ensure that an error in one module didn't bring down the whole system.

If you aren't worried about untrused code but do need to have each file totally isolated then you might be best off looking into various systems of interprocess communication. I've actually had some luck using Redis for this (which sounds ridiculous but actually has been very easy and effective).

Anyway hopefully some of that helps you. Without more information it's hard to provide more than general thoughts and a guide to better googling.

Community
  • 1
  • 1
Finn
  • 1,823
  • 2
  • 15
  • 31