3

ML / Tensorflow beginner.

Can any of these already-trained models be loaded on tfjs and re-trained there, then exported to Downloads or is Tensorflow python the only way to go?

I see this process is well described and documented in this tutorial for Tensorflow Python but unfortunately I can't find any documentation/tutorial to re-train an object detection model on the browser with tfjs (image classification yes, object detection no).

I see how I could load the coco-ssd model using npm, then probably even trigger saving it to downloads, but what about:

  • config file (need to modify it because I want to have only one class, not 90)
  • annotated images (both .jpg, .xml and .csv)
  • labels.pbtxt
  • .record files

Is there any way to go through the process of retraining an ssd model such as ssd_inception_v2_coco and I'm not hitting the right google keywords or is it just not possible in the current state of the framework?

avd
  • 45
  • 7
  • What do you mean by annotating images ? – edkeveked Feb 27 '19 at 18:37
  • The batch of images and their .xml with the bounding boxes values. – avd Feb 27 '19 at 20:22
  • You could win the bounty if you describe your solution (retrain model for TFjs) at https://stackoverflow.com/questions/55849309/. Any answer is welcome; the bounty otherwise forfeits. – serv-inc May 03 '19 at 11:57

1 Answers1

-1

You can use transfer learning by using coco-ssd model as a feature extractor. An example of transfer-learning can be seen here.

Here is a model which extracts features using a features extractor as an input for a new sequential model.

const loadModel = async () => {
  const loadedModel = await tf.loadModel(MODEL_URL)
  console.log(loadedModel)
  // take whatever layer except last output
  loadedModel.layers.forEach(layer => console.log(layer.name))
  const layer = loadedModel.getLayer(LAYER_NAME)
  return tf.model({ inputs: loadedModel.inputs, outputs: layer.output });
}
loadModel().then(featureExtractor => {
  model = tf.sequential({
    layers: [
      // Flattens the input to a vector so we can use it in a dense layer. While
      // technically a layer, this only performs a reshape (and has no training
      // parameters).
      // slice so as not to take the batch size
      tf.layers.flatten(
        { inputShape: featureExtractor.outputs[0].shape.slice(1) }),
      // add all the layers of the model to train
      tf.layers.dense({
        units: UNITS,
        activation: 'relu',
        kernelInitializer: 'varianceScaling',
        useBias: true
      }),
      // Last Layer. The number of units of the last layer should correspond
      // to the number of classes to predict.
      tf.layers.dense({
        units: NUM_CLASSES,
        kernelInitializer: 'varianceScaling',
        useBias: false,
        activation: 'softmax'
      })
    ]
  });
})

To detect a single object out of the 90 classes of coco-ssd, one could simply use a conditional test on the prediction of coco-ssd.

const image = document.getElementById(id)

cocoSsd.load()
  .then(model => model.detect(image))
  .then(prediction => {
if (prediction.class === OBJECT_DETECTED) {
  // display it the bbox to the user}
})

If the class does not exist in coco-ssd, then one needs to builds a detector.

edkeveked
  • 17,989
  • 10
  • 55
  • 93
  • I'm not sure I follow 100%. In the Pacman tfjs example a mobilenet model is created and later labels are passed every time a webcam shot happens. These labels are just a number from 0 to 3. In my case, can I load the cocoSSD model and then add to my controller dataset a predicted image that I load from assets folder along with its bounding boxes (name, width, height, x and y, a tensor5d)? I would add like that all 150 images of my custom object and then just train like in the example, then save the model to downloads? Please let me know if I understood right and thanks a lot for your help! – avd Feb 27 '19 at 20:21
  • Just saw your edit. I think I got it, but I'm still not sure about the bounding boxes, where would I be passing that data exactly? (name of the object in the image, positionX, positionY, width and height) – avd Feb 27 '19 at 20:38
  • @avd, if your goal is to predict a single object, you can use coco-ssd to predict the class of an image. With a conditional test on that class, you can select to display that object to the user or not. There is no need to retrain a model for it – edkeveked Feb 27 '19 at 20:59
  • My goal is to make it stop predict the 90 classes it predicts (I can filter with a conditional as you wrote) but I also want to add a custom additional class (let's say class 91) to detect with my own images (which were labeled in LabelImg making sure coco format is followed but I'm beginning to think that now that's not really needed anymore and I should add the images to the model through tfjs itself). – avd Feb 27 '19 at 21:17
  • transfer learning for ssd model is not clearly understood, the best option now is to retrain the TF model with your images. – Ping Yu Feb 28 '19 at 07:48
  • @PingYu what do you mean by the transfer learning is not clearly understood ? – edkeveked Feb 28 '19 at 07:53