1

I know that it's possible to send hyper-params as a dictionary to Trains.

But can it also automagically log hyper-params that are logged using the TF2 HParams module?

Edit: This is done in the HParams tutorial using hp.hparams(hparams).

Tensorboard HParams

DalyaG
  • 2,979
  • 2
  • 16
  • 19
Michael Litvin
  • 3,976
  • 1
  • 34
  • 40

1 Answers1

1

Disclaimer: I'm part of the allegro.ai Trains team

From the screen-grab, it seems like multiple runs with different hyper-parameters , and a parallel coordinates graph for display. This is the equivalent of running the same base experiment multiple times with different hyper-parameters and comparing the results with the Trains web UI, so far so good :)

Based on the HParam interface , one would have to use TensorFlow in order to sample from HP, usually in within the code. How would you extend this approach to multiple experiments? (it's not just automagically logging the hparams but you need to create multiple experiments, one per parameters set)

Wouldn't it make more sense to use an external optimizer to do the optimization? This way you can scale to multiple machines, and have more complicated optimization strategies (like Optuna), you can find a few examples in the trains examples/optimization.

Martin.B
  • 599
  • 3
  • 9
  • 1
    Thanks, I edited the question to emphasize that I was referring to automagically logging what's written to TB HParams when you do `hp.hparams(hparams)`. – Michael Litvin Aug 26 '20 at 08:30
  • 1
    So you mean for the call `hp.hparams(hparams)` to be automagically logged in Trains? Assuming hparams is a dict, it will save the trouble of calling `task.connect(hparams)`, is this correct? And a followup question, what happens if you have multiple tests (like in the screenshot) how would you separate one from the other ? – Martin.B Aug 26 '20 at 12:09
  • 1
    Yes I don't want to call `task.connect(hparams)` - because I'm not the one calling `hp.hparams(hparams)`, it's done by RayTune (similar to Optuna). Good question about separating multiple tests... I don't know. – Michael Litvin Aug 26 '20 at 13:50
  • 1
    I think solving the RayTune is key here. What we did with Trains and Optuna is we used Optuna for the Bayesian optimization, and Trains for the multi-node execution / running / parameter interface. How do you use RayTune? Are you seeing multiple experiments (i.e. is `Task.init` called multiple times)? – Martin.B Aug 26 '20 at 17:19
  • 1
    I haven't yet brought Trains and RayTune together. Used each separately until now. One of the next steps could be to start using them together. – Michael Litvin Aug 26 '20 at 21:37
  • 1
    Looks like my real question is somewhat more fundamental, so I've opened a new question focused on this. https://stackoverflow.com/questions/63606182/how-should-trains-be-used-with-hyper-param-optimization-tools-like-raytune – Michael Litvin Aug 26 '20 at 21:50