Tensorflow model training with a data set and additional predefined constraints

Question

I'm trying to understand how to create a tensorflow model which instead of doing a single classification would perform a multiple item classification at once.

For an example, I would like my tensorflow model to return me set of cloths depending on some weather conditions. The output should consist of set of clothes which wouldn't duplicate it's types (~how to correct the model when there is a obvious issue - winter jacket shouldn't be selected when the other cloth pieces are of lighter type):

good[0]: summer hat, light shirt, light pants, sandals (80%)
good[1]: summer hat, light shirt, normal pants, light shoes (10%)
good[2]: no hat, light shirt, normal pants, shoes (5%)
...
bad: summer hat, winter jacket, light shirt, sandals.

So far I was able to classify only single type of clothing ie: summer hat - 95% winter hat - 5%

To summarize my question: is it possible to create such model with tensorflow which can output a set of possible clothing variations without duplicated cloth types, and if yes - how to train that model?

Szymon Maszke · Accepted Answer · 2020-02-25T19:15:51.917

What you are trying to is called multilabel classification.

Basic multilabel classification

Model

Basically, your last layer should output N values where N is the number of clothes you have. Example model could look like this in tf2.0:

import tensorflow as tf

WEATHER_CONDITIONS = 15  # Based on 15 weather conditions
CLOTHES = 25  # Predict 25 possible clothes

model = tf.keras.models.Sequential(
    [
        tf.keras.layers.Dense(64, input_shape=(WEATHER_CONDITIONS,)),
        tf.keras.layers.Dense(128, activation="relu"),
        tf.keras.layers.Dropout(0.5),
        tf.keras.layers.Dense(256, activation="relu"),
        tf.keras.layers.Dense(CLOTHES),
    ]
)

Targets

Your targets should be binary vectors where each position corresponds to some predefined part of clothing, e.g. [0, 1, 0, 1, 0, 1] for corresponding [hat, shirt, jacket, sandals, normal pants, light pants] would mean shirt, sandals and light pants were predicted.

Loss function

Given the above, you should use tf.keras.losses.BinaryCrossEntropy with from_logits=True or tf.nn.sigmoid_cross_entropy_with_logits as loss functions so each vector value is in [0, 1] range (preferably the first one).

Correcting obvious issues

Easiest way would be to split your targets into multiple groups.

For example:

[summer hat, light shirt, light pants, sandals, light shoes]

[winter hat, shirt, pants, boots, shoes]

Now your model should output two targets: either 0 or 1 (whether it is a light or heavy clothing) and binary vector with 5 elements corresponding to specific item. This would require two losses and based on the predicted value specific group is chosen.

You may specify maximum penalty if wrong group is chosen and write your own custom loss function. You can check how to do that in this blogpost or this answer among may others here on StackOverflow.