Optuna pass dictionary of parameters from "outside"

Question

I am using Optuna to optimize some objective functions. I would like to create my custom class that "wraps" the standard Optuna code.

As an example, this is my class(it is still a work in progress!):

class Optimizer(object):
    
    def __init__(self, param_dict, model, train_x, valid_x, train_y, valid_y):
        self.model = model
        self.param_dict = param_dict
        self.train_x, self.valid_x, self.train_y, self.valid_y = train_x, valid_x, train_y, valid_y
        
    def optimization_function(self, trial):
        self.dtrain = lgb.Dataset(self.train_x, label=self.train_y)
        gbm = lgb.train(param, dtrain)
        
        preds = gbm.predict(self.valid_x)
        pred_labels = np.rint(preds)
        accuracy = sklearn.metrics.accuracy_score(self.valid_y, pred_labels)
        return accuracy
    
    
    def optimize(self, direction, n_trials):
        study = optuna.create_study(direction = direction)
        study.optimize(self.optimization_function, n_trials = n_trials)    
        return study.best_trial

I am trying to wrap all the "logic" of optuna optimization in this class, instead of writing everytime some code as the following one (from docs):

import optuna


class Objective(object):
    def __init__(self, min_x, max_x):
        # Hold this implementation specific arguments as the fields of the class.
        self.min_x = min_x
        self.max_x = max_x

    def __call__(self, trial):
        # Calculate an objective value by using the extra arguments.
        x = trial.suggest_float("x", self.min_x, self.max_x)
        return (x - 2) ** 2


# Execute an optimization by using an `Objective` instance.
study = optuna.create_study()
study.optimize(Objective(-100, 100), n_trials=100)

I would like to make my code "modular" and merge everything together in a single class. My final goal is to set different "templates" of optimization function, based on the given input model in the __init__ function.

So, getting back to the main question, I would like to pass from the outside the param dictionary. Basically I would like to be able to declare it from outside my class and pass my dictionary in the __init__ function.

However the ranges and distributions commonly used inside Optuna's codes, depends on the trial object, so I am not able to do something like:

my_dict = {
    'objective': 'binary',
    'metric': 'binary_logloss',
    'verbosity': -1,
    'boosting_type': 'gbdt',
     # HERE I HAVE A DEPENDENCY FROM trial.suggest_loguniform, I can't declare the dictionary outside the objective function
    'lambda_l1': trial.suggest_loguniform('lambda_l1', 1e-8, 10.0),
    'lambda_l2': trial.suggest_loguniform('lambda_l2', 1e-8, 10.0),
    'num_leaves': trial.suggest_int('num_leaves', 2, 256),
    'feature_fraction': trial.suggest_uniform('feature_fraction', 0.4, 1.0),
    'bagging_fraction': trial.suggest_uniform('bagging_fraction', 0.4, 1.0),
    'bagging_freq': trial.suggest_int('bagging_freq', 1, 7),
    'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
} 
my_optimizer = Optimizer(my_dict, ..., ..., ..., ......)
best_result = my_optimizer.optimize('maximize', 100)

Is there any work around or solution to pass this dictionary?

I have the same question, please do update here if you find a solution. Thank you — NIM4, Jul 08 '21 at 04:44

score 4 · Answer 1 · answered Apr 10 '22 at 11:50

I am not sure if I understand your question; but do you mean you want to pass a dict to objective function ?

if yes, this works for me, using lambda, from the faq of optuna:

import optuna

# Objective function that takes three arguments.
def objective(trial, min_x, max_x):
    x = trial.suggest_float("x", min_x, max_x)
    return (x - 2) ** 2


# Extra arguments.
min_x = -100
max_x = 100

# Execute an optimization by using the above objective function wrapped by `lambda`.
study = optuna.create_study()
study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100)

score 2 · Answer 2 · answered Sep 27 '22 at 21:36

Another faster way to do this, since you are trying to let objective take multiple arguments is to use python's built-in functools.partial

from functools import partial

def objective(trial, param1, param2):

     pass


param1 = 2
param2 = 3
objective = partial(objective, param1 = parma1, param2 = param2)
study.optimize(objective, n_trials = 100)

score 0 · Answer 3 · answered Oct 23 '21 at 10:48

How about this, create a detailed dict to be passed to the class, then reconstruct it for trial suggestion type.

code

class Optimizer(object):    
    def __init__(self, param_dict):
        self.param_dic = param_dict

        self.objective = self.param_dic.get('objective', None)
        self.metric = self.param_dic.get('metric', None)

    def optimization_function(self, trial):
        suggested_param = {}  # param storage

        # int
        int_param = self.param_dic['param'].get('int', None)
        if int_param is not None:     
            for k, v in int_param.items():
                suggested = trial.suggest_int(k, v['low'], v['high'])
                suggested_param.update({k: suggested})

        # log
        loguniform_param = self.param_dic['param'].get('loguniform', None)
        if loguniform_param is not None:
            for k, v in loguniform_param.items():
                suggested = trial.suggest_loguniform(k, v['low'], v['high'])
                suggested_param.update({k: suggested})

        a = suggested_param.get('a', None)
        b = suggested_param.get('b', None)
        c = suggested_param.get('c', None)

        return a + b + 1.5*c


    def optimize(self, direction, n_trials):
        study = optuna.create_study(direction = direction)
        study.optimize(self.optimization_function, n_trials = n_trials)
        return study.best_trial


my_dict = {
    'objective': 'binary',
    'metric': 'binary_logloss',
    'param': {
        'int': {
            'a': {
                'low': 0,
                'high': 20
            },
            'b': {
                'low': 0,
                'high': 10
            }
        },
        'loguniform': {
            'c': {
                'low': 1e-8,
                'high': 10.0
            }
        }
    }
}

my_optimizer = Optimizer(my_dict)
best_result = my_optimizer.optimize('maximize', 100)
print(f'best param: {best_result.params}')
print(f'best value: {best_result.values}')

J R · Answer 4 · 2022-07-03T09:28:32.367

I think I understand your question more now.

If I wanted to pass the params to the objective method from outside the class - i.e. not hardcoded into any part of the class, one way I could think of is to construct the params dict where the value is a string.

'min_child_weight': "@trial.suggest_int('min_child_weight', 1, 10)@"

Then from the objective function, remove all occurrences of "@ and @"

I've tried testing trial= optuna.trial.Trial and other permutations, but to no avail.

Optuna pass dictionary of parameters from "outside"

4 Answers4

code