Making hyperparameters add up to 1 when fine tuning using Optuna

Question

I have a function which looks like this:

def fine_tuning(x,y,model1,model2,model3,trial):
   pred1 = model1.predict(x)
   pred2 = model2.predict(x)
   pred3 = model3.predict(x)
   
   h1 = trial.suggest_float('h1', 0.0001, 1, log = True)
   h2 = trial.suggest_float('h1', 0.0001, 1, log = True)
   h3 = trial.suggest_float('h1', 0.0001, 1, log = True)

   pred = pred1 * h1 + pred2 * h2 + pred3 * h3

   return mean_absolute_error(y, pred)

The problem with this function is that h1+h2+h3 != 1. How would I change this function in order to make the sum of the hyperparmaters = 1?

Does this answer your question? [Generating a list of random numbers, summing to 1](https://stackoverflow.com/questions/18659858/generating-a-list-of-random-numbers-summing-to-1) — afsharov, Jul 23 '21 at 12:01
@afsharov Someone already answered the question, but deleted it for some reason — jr123456jr987654321, Jul 26 '21 at 08:42

score 1 · Accepted Answer · answered Jul 26 '21 at 09:04

Basically, you're looking for a dirichlet distribution for h1, 2, 3. Here's a guide on how to implement that for Optuna: https://optuna.readthedocs.io/en/latest/faq.html#how-do-i-suggest-variables-which-represent-the-proportion-that-is-are-in-accordance-with-dirichlet-distribution

Making hyperparameters add up to 1 when fine tuning using Optuna

1 Answers1