0

I'm at a complete loss with this one. I have implemented the following code, using the common pattern for defining a 'resolver' and the TPU Strategy.

However, when I run this, the tuner.search function is running at the speed of a regular CPU and I have no idea how to get it to utilise TPUs.

Really appreciate everyone's help!! :)

import keras_tuner as kt
import time

resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
strategy = tf.distribute.experimental.TPUStrategy(resolver)

def model_builder(hp):
    model = tf.keras.Sequential()

    hp_units = hp.Int('units', min_value=5, max_value=20, step=5)
    model.add(tf.keras.layers.Dense(units=hp_units, input_dim=X.shape[1], activation="relu"))
    model.add(tf.keras.layers.Dense(10))

    hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])

    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=hp_learning_rate), 
        loss="binary_crossentropy", 
        metrics=["accuracy"])

    return model

#with strategy.scope():
tuner = kt.Hyperband(
    model_builder, 
    objective="accuracy", 
    max_epochs=10, 
    factor=3, 
    distribution_strategy=strategy)

# Early Stopping Callback
#stop_early = tf.keras.callbacks.EarlyStopping(monitor="val_loss", patience=5)

tic = time.time()
tuner.search(X, Y, epochs=50, callbacks=[tf.keras.callbacks.EarlyStopping(patience=1)])
toc = time.time()

search_time = toc - tic
Larry125
  • 25
  • 1
  • 4
  • you know about the all ''google cloud bucket'' problem right? With TPUs you can't have the data just where you want – BestDogeStackoverflow Jan 02 '22 at 16:58
  • Hi :) - I know nothing about the "google cloud bucket" problem. What do you mean? – Larry125 Jan 02 '22 at 20:12
  • to be honest i don't know if things have changed or i am just dead wrong... what i remember is that to use the TPUs on google colab you need to host the data on the buckets of google cloud, if you try to run a NN on TPU with the data that you just loaded to the session it will not work – BestDogeStackoverflow Jan 03 '22 at 00:34
  • example: https://stackoverflow.com/questions/67063455/google-cloud-storage-for-google-colab-tpu-pricing – BestDogeStackoverflow Jan 03 '22 at 00:35
  • https://stackoverflow.com/questions/58009029/tf1-14tpucan-not-use-custom-tfrecord-dataset-on-colab-using-tpu https://github.com/tensorflow/tensorflow/issues/32651 – BestDogeStackoverflow Jan 03 '22 at 00:39
  • if i am not wrong this would explain why your model works but it's not using the TPU, and thats because it does not accept files in the session filesystem and it just goes back on using the CPU. – BestDogeStackoverflow Jan 03 '22 at 00:43

0 Answers0