0

I'm currently writing code in Python 2.7, which involves creating an object, in which I have two class methods and other regular methods. I need to use this specific combination of methods because of the larger context of the code I am writing- it's not relevant to this question, so I won't go into depth.

Within my __init__ function, I am creating a Pool (a multiprocessing object). In the creation of that, I call a setup function. This setup function is a @classmethod. I define a few variables in this setup function by using the cls.variablename syntax. As I mentioned, I call this setup function within my init function (inside the Pool creation), so these variables should be getting created, based on what I understand.

Later in my code, I call a few other functions, which eventually leads to me calling another @classmethod within the same object I was talking about earlier (same object as the first @classmethod). Within this @classmethod, I try to access the cls.variables I created in the first @classmethod. However, Python is telling me that my object doesn't have an attribute "cls.variable" (using general names here, obviously my actual names are specific to my code).

ANYWAYS...I realize that's probably pretty confusing. Here's some (very) generalized code example to illustrate the same idea:

class General(object):
    def __init__(self, A):
        # this is correct syntax based on the resources I'm using,
        # so the format of argument isn't the issue, in case anyone
        # initially thinks that's the issue
        self.pool = Pool(processes = 4, initializer=self._setup, initargs= (A, )

    @classmethod
    def _setup(cls, A):
        cls.A = A

    #leaving out other functions here that are NOT class methods, just regular methods

    @classmethod
    def get_results(cls):
        print cls.A

The error I'm getting when I get to the equivalent of the print cls.A line is this:

AttributeError: type object 'General' has no attribute 'A'

edit to show usage of this code: The way I'm calling this in my code is as such:

G = General(5)
G.get_results()

So, I'm creating an instance of the object (in which I create the Pool, which calls the setup function), and then calling get_results.

What am I doing wrong?

  • 1
    Did you ever call `General._setup`? – chepner Jul 27 '16 at 16:53
  • 3
    Particularly, did you ever call `General._setup` in the process that called `General.get_results`? – user2357112 Jul 27 '16 at 16:55
  • @chepner I'm calling it in when I'm creating the Pool object in my init function. –  Jul 27 '16 at 17:00
  • @user2357112 same as above^ –  Jul 27 '16 at 17:03
  • Please [edit] your question and show example(s) of code that causes the error—I would assume by making calls to the classmethods. – martineau Jul 27 '16 at 17:32
  • The class variable won't exist until `_setup()` has been called to create them. – martineau Jul 27 '16 at 17:50
  • @martineau Right. As I said in response to the above comments, the setup function is called in the creation of the Pool. That's called in the __init__ function, which is called when I create an instance of the object, which is before the get_results function is called. –  Jul 27 '16 at 17:55
  • It's important to put information relevant to the question in it so folks don't have to read every comment to determine what you're asking and the problem is. – martineau Jul 27 '16 at 18:00
  • 1
    "I call this setup function within my init function" - no you don't. – user2357112 Jul 27 '16 at 18:04
  • @martineau Did you read the entire question? I answer that in the problem description in the 2nd paragraph. On top of that, if you look at the example code I gave, you would see for yourself in the __init__ function that I call the setup function in the Pool initialization. I deliberately took out everything not relevant to this question in my code example to make it easier to read. It's the only line in my init function, so I figured it would be read, if someone opted not to read my problem description. You didn't need to read the comments to get this info. You just had to read the question. –  Jul 27 '16 at 18:06
  • @user2357112 I apologize if that wording if confusing. I mention earlier in that paragraph that I call the setup function within the creation of the Pool object. The sentence you're quoting is just me paraphrasing what I had already said to avoid repeating the same sentence. But I'll edit it to make it less confusing, since I think I've caused a lot of confusion. –  Jul 27 '16 at 18:08
  • Ria: Yes, I read your question. That line in your `__init__()` method does _not__ call the static `setup()` method for the class in main process, only within each subprocess in the `Pool` it creates—so class attribute `A` doesn't exist in the main process when `get_results()` is called from within it. As you said yourself, your question is pretty confusing given all the omitted code confounded with your own misunderstandings about how multiprocessing works. – martineau Jul 29 '16 at 15:52

1 Answers1

1

The reason General.A does not get defined in the main process is that multiprocessing.Pool only runs General._setup in the subprocesses. This means that it will not be called in the main process (where you call Pool).

You end up with 4 processes where in each of them there is General.A is defined, but not in the main process. You don't actually initialize a Pool like that (see this answer to the question How to use initializer to set up my multiprocess pool?)

You want an Object Pool which is not natively impemented in Python. There's a Python Implementation of the Object Pool Design Pattern question here on StackOverflow, but you can find a bunch by just searching online.

Community
  • 1
  • 1
Artyer
  • 31,034
  • 3
  • 47
  • 75
  • @Ria You may be missing some functions. That website says that a standalone repo is available [here](https://github.com/aholmberg/driver-multiprocessing) – Artyer Jul 27 '16 at 18:15
  • @Ria: Yeah, you followed it wrong. The examples in that link never try to access anything set up in the pool `initializer` from the main process, because that initialization never happens in the main process. – user2357112 Jul 27 '16 at 18:18
  • @Artyer Thanks for your help. I left out a lot of functions in my code example because I thought they would just make things more confusing, and so I do think I have all of them in my actual code. But it's very possible I missed something. Thanks again! –  Jul 27 '16 at 18:21
  • @user2357112 Doesn't the example from the link call cls.session and cls.prepared in the get_results function? And those variables are set up in the setup function. –  Jul 27 '16 at 18:23
  • @Ria: Only in the single-process version. The multi-process version makes sure never to do that. – user2357112 Jul 27 '16 at 18:26
  • @Ria: You're reading it completely wrong. Only the processes in the process pool access anything initialized in `_setup`. – user2357112 Jul 27 '16 at 18:33
  • @Ria: You still have to call that from a process where `cls.session` and `cls.prepared` actually exist, a.k.a. not the master process. – user2357112 Jul 27 '16 at 18:45