1

I am using keras tuner in a Deep Learning project to fit hyperparameters.
Because there are many combinations of hyperparameters, even with a GPU it takes more than 24 hours to run. To have reproductible results, we fixed the random state (seed of numpy and tensorflow) before running the training.

I would like to know if there are an easy way to divide the tuning search into several steps so that we can run it each night (e.g divide it into 3 parts which takes each 8 hours to run). Thus, if there are 300 trials, we would like to run the first 100 trials the first night, trials 100 to 200 the second night, and finally the last 100 trials the last night.
I have try to search how to have the configuration of a particular trial_id (with the function populate_state of kera_tuner.tuners.random_search) so that I run a training on that configuration but I didn't find out how to use it.

Thanks in advance for your answers.

baptiste_new
  • 11
  • 1

1 Answers1

0

maybe I did not understood the question, but if problem is time comsumption , why not use hyperband instead of random search ?

EDIT : also there is a chapter in link about distributed tuning . Or you could split your hyperparameters list into 3 or 4 HyperParameters object and run them one per night (like the example in link

Chmick Chmick
  • 43
  • 1
  • 6
  • I have thought to this solution, it could be interesting nevertheless the convergence of our model is quite slow (takes several dozen epochs at least, and the loss continues to decrease stricly so early stoping might not be relevant in our case) and in addition to that, the speed of the convergence depends on the hyperparameters of the model we train (some models learn quicker but are less performant in the end) – baptiste_new Aug 19 '21 at 14:23
  • @baptiste_new before uisng keras tuner , i was looping through different paremeters manually. To gain some speed compare to my hone graphic card GeForce GTX 1060 6GB, the only solution I found was to rent some machine on AWS (at least a p3.2xlarge). If you convergence is slow maybe your learning rate is to small. I also use early stopping but i put a big number in epochs parameter – Chmick Chmick Aug 19 '21 at 16:40
  • also there is a chapter in https://blog.tensorflow.org/2020/01/hyperparameter-tuning-with-keras-tuner.html – Chmick Chmick Aug 19 '21 at 16:53
  • Thank you for your answer. We can't run our training on big data solution currently :(, and I saw that hyperparameter distribution could be interesting but I have only one GPU, idk if it could be distributed on one worker for different days. I think I will split the hyperparameters space manually it would be easier ahah. – baptiste_new Aug 20 '21 at 07:56
  • BTW . AWS may have a free credits program to test your code on their machine . You have to create an aws account , but it's worth the shot – Chmick Chmick Aug 22 '21 at 16:45