I am trying to train an object detection model using ImageAI library and here is my code.
from imageai.Detection.Custom import DetectionModelTrainer
trainer = DetectionModelTrainer()
trainer.setModelTypeAsYOLOv3()
trainer.setDataDirectory(data_directory='Datasets/hololens')
trainer.setTrainConfig(object_names_array=['hololens'], batch_size=4, num_experiments=100,
train_from_pretrained_model='pretrained-yolov3.h5')
trainer.trainModel()
However, the training just keeps stopping/pausing/hanging halfway. It would stop right after showing an error saying "Profiler session tear down" as shown below:
2/480 [..............................] - ETA: 4:51 - loss: 183.8092 - yolo_layer_loss: 25.4697 - yolo_layer_1_loss: 50.0500 - yolo_layer_2_loss: 96.7067 2021-07-30 14:32:31.192212: I tensorflow/core/profiler/lib/profiler_session.cc:71] Profiler session collecting data. 2021-07-30 14:32:31.196004: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1487] CUPTI activity buffer flushed 2021-07-30 14:32:31.449851: I tensorflow/core/profiler/internal/gpu/cupti_collector.cc:228] GpuTracer has collected 5713 callback api events and 5681 activity events. 2021-07-30 14:32:31.558638: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down
I'm using GTX 1080Ti, Tensorflow 2.4.1, CUDA v11. I've been training image classification models completely fine but when I try to run this object detection model training, this problem happens.
Thank you in advance! :)
