Is it possible to create constraints in parallel?

Question

I have a relatively simple Pyomo model, where I wish to create the constraints in parallel. I have created the following functions:

def constraintA():
    constraintA = pyo.ConstraintList()
    for y, h in model.YH:
        constraintA.add(model.varA[y, h] <= data[y, h])
    return constraintA

def constraintB():
    constraintB = pyo.ConstraintList()
    for y, h in model.YH:
        constraintB.add(model.varB[y, h] >= other_data[y, h])
    return constraintB

def constraintC():
    constraintC = pyo.ConstraintList()
    for y, h in model.YH:
        constraintC.add(model.varC[y, h] <= data[y,h] * 0.5)
    return constraintC

Note that the model, data and other_data are considered 'global' and therefore can be used within these constraint functions. Now, what I would like to do is create attributes in the pyomo model with the respective constraints, but then in parallel.

So I start with:

constraints = [constraintA, constraintB, constraintC]
with ProcessPoolExecutor() as pool:
    for work_item in enumerate(constraints):
        fs = [pool.submit(c) for c in constraints]

However, I get "finished raised AttributeError". Does this mean that despite Python only has to read and copy it, it is locking the object and therefore cannot access the attribute and therefore is unable to create the constraints?

In order to resolve this issue, I was wondering whether I need to explicitly copy the model and the data and other_data to the constraint functions themselves. Or is there another way to achieve this, without "passing variables around"?

Can I inquire about your motivation to do this? In almost any reasonably sized model, the construction time is trivial compared to the solve time... Also, you realize that the constraint lists you are attempting to create are not part of the model, and attaching them in your functions with `model.constraintA = ...` will likely make the threads co-dependent. — AirSquid, Sep 12 '22 at 22:03
I just wish to speed up the model building, at some point indeed I will have to attach them to the model, however, I think attaching them needs to be done sequentially, while constructing the constraintList, can be done in parallel. — Snowflake, Sep 13 '22 at 05:13
In addition, I seem to be experiencing the following as described here: https://stackoverflow.com/questions/51269351/pyomo-seems-very-slow-to-write-models, however, for readability, I wish to create the constraintList using Pyomo's apprach rather than manually crafing the constraintList. — Snowflake, Sep 13 '22 at 05:28

score 2 · Answer 1 · answered Sep 13 '22 at 13:04

For all intents and purposes, Pyomo model construction is a serial exercise.

This is primarily due to the CPython GIL (Global Interpreter Lock), which effectively serializes everything in Python (multithreading in CPython is actually serial, and relies on things like I/O waits to provide "parallelism"). In theory, you can construct model fragments in distributed processes (either through forked processes or something like mpi4py). The problem is that the process of serializing the model fragments and collecting them back into a single process is actually more costly than just generating the model serially in the single process.

There are some Pyomo-based ideas that may enable parallel model construction in the future: one is possibly supporting distributed model construction as well as LP / NL file creation, then merging the LP / NL file fragments instead of bringing back the original modeling objects, and the other would rely on a compiled C extension that would accept a Constraint "template" and construct objects on the C side (where we can avoid the GIL). Both of these are very experimental ideas at the moment.

There is also always the possibility that CPython will remove the GIL (in which case Pyomo will almost certainly immediately add multithreading to constraint construction). There have been many efforts over the years to remove the GIL from CPython, and maybe one will finally succeed and be adopted by CPython.

The last thing to point to is @AirSquid's comment that model construction is usually not the bottleneck. If it is, then some time looking at how you are defining your model / constraints would be warranted (Pyomo is very forgiving in what it will allow you to do, but not all approaches are equally performant).

Is it possible to create constraints in parallel?

1 Answers1