Keras iterator with augmented images and other features

Question

Say you have a dataset that has images and some data in a .csv for each image. Your goal is to create a NN that has a convolution branch and another one (in my case an MLP).

Now, there are plenty of guides (one here, another one) on how to create the network, that's not the problem.

The issue here is how do I create an iterator in the form of [[convolution_input, other_features], target] when the convolution_input is from a Keras ImageDataGenerator flow that adds augmented images.

More specifically, when the nth image (that may be an augmented one or not) is fed to the NN, I want its original features inside other_features.

I found a few attempts (here and here, the second one looked promising but I wasn't able to figure out how to handle augmented images) in doing exactly that but they do not seem to take into account the possible dataset manipulation that the Keras generator does.

Question: are you ok with `flow` or do you need `flow_from_directory`? (`flow` means you can keep all images loaded in memory) — Daniel Möller, Mar 06 '20 at 12:52
Well, I just want a flow that automatically handles image transformation. In my case I was using `flow_from_dataframe` since I have file names, features and classes — Lamberto Basti, Mar 09 '20 at 01:20

score 5 · Accepted Answer · edited Nov 28 '21 at 21:03

Let's say, you have a CSV, such that your images and the other features are in the file.

Where id represents the image name, and followed by the features, and followed by your target, (class for classification, number for regression)

|         id          | feat1 | feat2 | feat3 | class |
|---------------------|-------|-------|-------|-------|
| 1_face_IMG_NAME.jpg |   1   |   0   |   1   |   A   |
| 3_face_IMG_NAME.jpg |   1   |   0   |   1   |   B   |
| 2_face_IMG_NAME.jpg |   1   |   0   |   1   |   A   |
|         ...         |  ...  |  ...  |  ...  |  ...  |

First, let us define a data generator, and later we can override it.

Let us read the data from the CSV in a pandas data frame and use keras's flow_from_dataframe to read from the data frame.

df = pandas.read_csv("dummycsv.csv")
datagen = ImageDataGenerator(rescale=1/255.)
generator = datagen.flow_from_dataframe(
                df,
                directory="out/",
                x_col="id",
                y_col=df.columns[1:],
                class_mode="raw",
                batch_size=1)

You can always add your augmentation in ImageDataGenerator.

Things to note in the above code in flow_from_dataframe is

x_col = the image name

y_col = typically columns with the class name, but let us override it later by first providing all the other columns in the CSV. i.e. feat_1, feat_2.... till class_label

class_mode = raw, suggests the generator to return all the values in y as is.

Now let us override/inherit the above generator and create a new one, such that it returns [img, otherfeatures], [target]

Here is the code with comments as explanations:

def my_custom_generator():
    # to keep track of complete epoch
    count = 0 
    while True:
        if count == len(df.index):
            # if the count is matching with the length of df, 
            # the one pass is completed, so reset the generator
            generator.reset()
            break
        count += 1
        # get the data from the generator
        data = generator.next()

        # the data looks like this [[img,img] , [other_cols,other_cols]]  based on the batch size        
        imgs = []
        cols = []
        targets = []

        # iterate the data and append the necessary columns in the corresponding arrays 
        for k in range(batch_size):
            # the first array contains all images
            imgs.append(data[0][k])
      
            # the second array contains all features with last column as class, so [:-1]
            cols.append(data[1][k][:-1])

            # the last column in the second array from data is the class
            targets.append(data[1][k][-1])

        # this will yield the result as you expect.
        yield [imgs,cols], targets

Create a similar function for your validation generator. Use train_test_split to split your data frame if you need it and create 2 generators and override them.

Pass the function in model.fit_generator like this

model.fit_generator(my_custom_generator(),.....other params)

But how can `if(count==len(df.index))` keep track of the epoch if the augmented dataset is far more numerous then the original one? — Lamberto Basti, Mar 07 '20 at 15:24
Augmentations are rndomly applied to images. It will not increase the number of images unless you save them separately and use it as an unique instance in the training set. How augmentation helps is during each epoch different augmentation is applied randomly thereby making it look like different images — venkata krishnan, Mar 07 '20 at 17:07
Wow, that is a huge information. I'd recommend to add it to the answer as well. Also, why such important information is not written explicitly in the Keras documentation? — Lamberto Basti, Mar 07 '20 at 19:50
Even i figured it myself, because when we mention steps per epoch in the training, we generally divide the training length by batch size which means, in an epoch, it passes all the images only once. I hope i am right.. — venkata krishnan, Mar 08 '20 at 08:10
@venkatakrishnan I built similar generator but it doesn't work with error about tuple. Could you please see my question https://stackoverflow.com/questions/62744659/tuple-object-has-no-attribute-rank-when-fit-keras-model-with-custom-generato? — feeeper, Jul 05 '20 at 18:49

Keras iterator with augmented images and other features

1 Answers1

Linked