How to extract Indexed Variable information in pyomo Model and build Pandas Dataframe

Question

I am attempting to solve a MIP using Pyomo with several Indexed Variables of different sizes. What I would like to do is export the results of the variables to a dataframe so that I can further analyze these results. My problem is that I cannot figure out a way to nicely automate the process as the size of the Indexed Variables can change in each model simulation.

Here is an example of some variables I would create:

import pyomo.environ as pyo

model = pyo.ConcreteModel()

model.T = pyo.RangeSet(0, 10)
model.Generators = pyo.Set(initialize=(['equip_1', 'equip_2']))
model.Storages = pyo.Set(initialize=(['storage_1', 'storage_2']))

model.var_1 = pyo.Var(model.T, model.Generators, domain=pyo.NonNegativeReals)
model.var_2 = pyo.Var(model.T, domain=pyo.NonNegativeReals)
model.var_3 = pyo.Var(model.T, model.Storages, domain=pyo.NonNegativeReals)
model.var_4 = pyo.Var(model.T, model.Generators, domain=pyo.Binary, initialize=0)

# constraints and objective function here, which I don't think are relevant for the question
.
.
.
SolverFactory('cbc').solve(model).write()

Now, I want to create the dataframe with model.T as the index and the variable name plus the model.Generator or model.Storages as a multiIndex column (I am assuming it would have to be a multiIndex, but perhaps not). A crude example of how I want it to look is shown below:

  |      Var_1      | Var_2 |        Var_3        |      Var_4
  | equip_1 equip_2 | None  | storage_1 storage_2 | equip_1 equip_2
m |   0    |   0    |   0   |     0     |   0     |   0    |   1  
o |   1    |   1    |   1   |     1     |   1     |   1    |   0  
d |   2    |   2    |   2   |     2     |   2     |   0    |   1  
e |   3    |   3    |   3   |     3     |   3     |   1    |   0  
l |   4    |   4    |   4   |     4     |   4     |   0    |   1  
. |   5    |   5    |   5   |     5     |   5     |   1    |   0  
T |   6    |   6    |   6   |     6     |   6     |   0    |   1

The values shown are just examples. Also, the model.T doesn't have to be the index as it would correlate to a standard index created with a dataframe.

I have been pulling my hair out trying to get this to work but I am struggling to find the right syntax in pyomo. I feel there must be an easy way to extract this data as I can't be the first one trying to do this but I've searched the internet to no avail for this specific problem. The problem I have is trying to extract the variable index (model.T and model.Generators/Storages) from the variable results in an easy loop or vectorized fashion.

Please! Any suggestions would be greatly appreciated. Let me know if I haven't made this clear enough.

score 7 · Accepted Answer · answered May 11 '21 at 21:06

Totally doable...

First, let's cover a basic example of how to (in the general sense) get the results out of pyomo after a solve. Recall, that after the solver completes, the optimal values of the variables will be loaded into the model variables. Here is a cheap example that shows 3 ways to extract data from the variables. There are many others, depending on what needs to be done. Of note, the last method here, converting to a dictionary, is key to transitioning over to pandas because we all know pandas love dictionaries.

# extracting values 1
import pyomo.environ as pyo
import pandas as pd

m = pyo.ConcreteModel('value extraction')

m.S = pyo.Set(initialize=range(3))
m.X = pyo.Var(m.S, initialize={0:-1, 1:2.5, 2:12})   # just initializing to simulate solving

# show all of X
m.X.display()

# iterate through and show values
print()
for s in m.S:
    print(f'for index {s} X[{s}] is: {m.X[s].value}')

# dump into a dictionary.... an entry point for pandas!
print()
print(m.X.extract_values())

# make a pd.Series indexed by the index set(s)
print()
x_vals = pd.Series(m.X.extract_values(), name=m.X.name)
print(x_vals)

Yields:

X : Size=3, Index=S
    Key : Lower : Value : Upper : Fixed : Stale : Domain
      0 :  None :    -1 :  None : False : False :  Reals
      1 :  None :   2.5 :  None : False : False :  Reals
      2 :  None :    12 :  None : False : False :  Reals

for index 0 X[0] is: -1
for index 1 X[1] is: 2.5
for index 2 X[2] is: 12

{0: -1, 1: 2.5, 2: 12}

0    -1.0
1     2.5
2    12.0
Name: X, dtype: float64

Part II:

Ok, on to building a more comprehensive solution to your challenge of mashing the whole result into a multi-indexed dataframe. Of note, I only initialized the values of the variables in this to simulate solving. Normally, not needed for variables.

So this builds on the concept before, which is that a pd.Series is just an array that is indexed by an index array, and can be constructed from dictionaries. I'm not the slickest pandas operator, so there may be some way to combine some of the pd commands, but who cares? This is step-by-step and if you are dealing with variables from an optimization, it is not going to be too huge, so tweaking is unnecessary.

# extracting values 2
import pyomo.environ as pyo
import pandas as pd

model = pyo.ConcreteModel()

model.T = pyo.RangeSet(0, 10)
model.Generators = pyo.Set(initialize=(['equip_1', 'equip_2']))
model.Storages = pyo.Set(initialize=(['storage_1', 'storage_2']))

model.var_1 = pyo.Var(model.T, model.Generators, initialize=1.5, domain=pyo.NonNegativeReals)
model.var_2 = pyo.Var(model.T, initialize=2.5, domain=pyo.NonNegativeReals)
model.var_3 = pyo.Var(model.T, model.Storages, initialize=3.5, domain=pyo.NonNegativeReals)
model.var_4 = pyo.Var(model.T, model.Generators, domain=pyo.Binary, initialize=0)

#model.display()

# let's convert each var to a pandas series, indexed by model.T...

# get all the variables (assuming the fuller model will have constraints, params, etc.)
model_vars = model.component_map(ctype=pyo.Var)


serieses = []   # collection to hold the converted "serieses"
for k in model_vars.keys():   # this is a map of {name:pyo.Var}
    v = model_vars[k]

    # make a pd.Series from each    
    s = pd.Series(v.extract_values(), index=v.extract_values().keys())

    # if the series is multi-indexed we need to unstack it...
    if type(s.index[0]) == tuple:  # it is multi-indexed
        s = s.unstack(level=1)
    else:
        s = pd.DataFrame(s)         # force transition from Series -> df
    #print(s)

    # multi-index the columns
    s.columns = pd.MultiIndex.from_tuples([(k, t) for t in s.columns])

    serieses.append(s)

df = pd.concat(serieses, axis=1)
print(df)

Yields

     var_1         var_2     var_3             var_4        
   equip_1 equip_2     0 storage_1 storage_2 equip_1 equip_2
0      1.5     1.5   2.5       3.5       3.5       0       0
1      1.5     1.5   2.5       3.5       3.5       0       0
2      1.5     1.5   2.5       3.5       3.5       0       0
3      1.5     1.5   2.5       3.5       3.5       0       0
4      1.5     1.5   2.5       3.5       3.5       0       0
5      1.5     1.5   2.5       3.5       3.5       0       0
6      1.5     1.5   2.5       3.5       3.5       0       0
7      1.5     1.5   2.5       3.5       3.5       0       0
8      1.5     1.5   2.5       3.5       3.5       0       0
9      1.5     1.5   2.5       3.5       3.5       0       0
10     1.5     1.5   2.5       3.5       3.5       0       0

Thank you so much @AirSquid this is perfect! You just made my week. — Corey Duncan, May 14 '21 at 11:29

How to extract Indexed Variable information in pyomo Model and build Pandas Dataframe

1 Answers1

Yields:

Part II:

Yields

Linked