2

I am using Ray Tune and I am disappointed by the lack of options for conditional / nested hyperparameters. It seems I will have to hack something together, but since I can't be the first one who had this problem I'm wondering how other people solved it.

Say that my algorithm has a baseline mode as well as an advanced mode, and the advanced mode has two parameters. This gives a total of 3 parameters.

mode: ['baseline', 'advanced']
param_a: [1,2]
param_b: [5,10]

Since param_a and param_b are only used by the advanced mode, I would like it to run with the following combination of parameters:

baseline, None, None
advanced, 1, 5
advanced, 1, 10
advanced, 2, 5
advanced, 2, 10

Note that the baseline is only used once.

What's the easiest way to get this to work in Ray Tune?

Is there a way to extend it for nested conditionals? This example has only one conditional, but more complex trials could have several.

It should also be possible to sample parameters from a random distribution if and only if those parameters should actually be used.

The suggested solution in their FAQ is to basically build the entire search space myself if I have conditions like this. There must be an easier way to achieve this.

Florian Dietz
  • 877
  • 9
  • 20
  • I'm facing similar problems (especially regarding nested and not just conditional hyperparameters). Probably you've read [this](https://docs.ray.io/en/latest/tune/tutorials/tune-search-spaces.html#how-to-use-custom-and-conditional-search-spaces). But for others: Some of the requested functionality may be achieved by using `tune.sample_from`. But mind that you will lose access to all »smart« optimizers (only random and grid search are compatible with `tune.sample_from`). – user3389669 Sep 08 '22 at 14:10

0 Answers0