0

I am trying to extract the results of a pyomo instance, with several indexed expressions and params. The index size is around 675 and there are 13 indexed expressions. The expressions involve either sums or products. I'm using the following code to extract the numerical results but extracting the results is taking longer than solving the model.


from pyomo.core import ConcreteModel, Expression, Param, value, Var
import pandas as pd

def extract_values_from_instance(instance: ConcreteModel)
    model_attributes = {
        "model_vars": instance.component_map(ctype=Var),
        "model_param": instance.component_map(ctype=Param),
        "model_expr": instance.component_map(ctype=Expression),
    }
    LOGGER.info('Extracting model values')
    list_of_series = []
    scalars_dict = {}
    for values in model_attributes.values():
        for key, var in values.items():
            if var.is_indexed():
                list_of_series.append(
                    pd.Series(
                        {k: value(v) for k, v in var.extract_values().items()},
                        index=var.extract_values().keys(),
                        name=key,
                        dtype="float64",
                    )
                )
            else:
                scalars_dict[key] = var()

    return pd.concat(list_of_series, axis=1), scalars_dict

Is there a way to extract the numerical results or at least calculate the numerical value from the numexpr that it does not involve calling value() for each item?

UPDATE:

The culprit seemed to be an expression of the following format

def distance_traveled_rule(model, current_time) -> float:
    if current_time == model.time_start.value:
        return model.initial_distance # Param
    distance_traveled = (
        model.initial_distance # Param
        + sum(              
            model.velocity[time_stamp] * interval
            for time_stamp in model.velocity
            if time_stamp < current_time
        )
    )
    return distance_traveled
model.distance_traveled = Expression(
    model.velocity.index_set(),
    rule=distance_traveled_rule,
    doc="Distance traveled at each time stamp",
)
poeticcapybara
  • 555
  • 1
  • 5
  • 15
  • 1
    This seems like a near duplicate of: https://stackoverflow.com/questions/67491499/how-to-extract-indexed-variable-information-in-pyomo-model-and-build-pandas-data – Bethany Nicholson Aug 02 '23 at 20:59

1 Answers1

1

@Bethany Nicholson: Looked kind of familiar!

Two things come to mind. not sure what your criteria is for quicker...

  1. Be sure you are using the latest version of Pyomo and a recent version of python. Both are significantly quicker at iterating over dictionaries etc. with dict_view usage, etc.

  2. You can eliminate some of the redundant dictionary creation you are doing in your loop. Right now in each series creation you are creating 3 dictionaries where 1 will do. (The comprehension of the extract_values (2 creations) and the second extract_values for a total of 3.). You can just use extract_values once, get a dictionary of what you want and the keys to the series are implied.

Try this mod to your loop, which is basically trimming around the edges a bit:

from pyomo.core import ConcreteModel, Expression, Param, value, Var, Set
import pandas as pd

def extract_values_from_instance(instance: ConcreteModel):
    model_attributes = {
        "model_vars": instance.component_map(ctype=Var),
        "model_param": instance.component_map(ctype=Param),
        "model_expr": instance.component_map(ctype=Expression),
    }
    # LOGGER.info('Extracting model values')
    list_of_series = []
    scalars_dict = {}
    for values in model_attributes.values():
        for key, var in values.items():
            if var.is_indexed():
                list_of_series.append(
                    pd.Series(
                        var.extract_values(),
                        name=key,
                        dtype="float64",
                    )
                )
            else:
                scalars_dict[key] = var()

    return pd.concat(list_of_series, axis=1), scalars_dict

m = ConcreteModel()

m.S = Set(initialize=range(10,15))
m.X = Var(m.S, initialize=2.0)

print(extract_values_from_instance(m))
AirSquid
  • 10,214
  • 2
  • 7
  • 31
  • Regarding 1. I'm currently using 6.6.1 so nothing to improve there but I probably should have added it in the post what I meant with quicker. Right now the optimization solve is taking 1-2s and the data extraction around 4-5s. I've found the culprit of the long times to be two expressions whose values are a cumulative sum of a variable and each was taking around 2s, while all other ones just tens of milliseconds. I still find it surprising that it takes so long. I've updated the original post with an example. And thanks for the extraction values snippet! – poeticcapybara Aug 03 '23 at 20:17
  • @poeticcapybara. I don't see any edits to your post (yet)? Are you using the expressions inside of the model for something or are you solely creating them to get a value? I've found that the *construction* of long expressions can be non-negligible, so if they aren't needed in the model, I would consider just computing the expression value from the variable outputs rather than making the expression in the first place. – AirSquid Aug 03 '23 at 20:27
  • We reached the same conclusion and we are now computing it outside but had to keep one of the two in the model, as we still use it in a constraint though. – poeticcapybara Aug 03 '23 at 20:33