fit a keras model with mixed input of type BatchDataset and numpy array

Question

I made a model that receives two input. When I fit the model with two numpy array it works. Here's an example:

model.fit(x=[image_input, other_features], y = y, epochs=epochs)

However, my problem is that other_features is a numpy array and image_input is loaded with keras using tf.keras.preprocessing.image_dataset_from_directory. The problem I face is:

1. how to give y correctly from the image_input? when I trained the model with only one input image_input, y is packed inside it so I didn't have to specify it in another parameters.
2. how to put a BatchDataset with numpy.array? when I did it anyway I received the error:

ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of types {"<class 'tensorflow.python.data.ops.dataset_ops.BatchDataset'>", "<class 'numpy.ndarray'>"}), <class 'NoneType'>

Answer 1

Okay I was able to solve it. I will write the detailed solution because I saw similar question to this posted many times with no answers. This is mixed input and the solution is to rely on a custom generator.

the first step is to make the custom generator. You have to return a list/dict of your inputs + output. I followed this link to make mine. Here's a sample code of my generator:

def generator(subset, batch_size=256):
    i = 0
    DIR = f"data/{subset}"
    image_files = pd.read_csv(f"{DIR}.csv")
    while True:
        batch_x = [list(), list()] # I have two input: image + feature vector
        batch_y = list() # output
        for b in range(batch_size):
            if i == len(image_files):
                i = 0
            filename = image_files.loc[i, "filename"]
            label = image_files.loc[i, "Class"]
            image_file_path = f'{DIR}/{label}/{filename}'
            i += 1
            image = cv2.imread(image_file_path, 0)
            batch_x[0].append(image)
            feat = get_feature_vector(filename)
            batch_x[1].append(feat)
            batch_y.append(one_hot(label))

        batch_x[0] = np.array(batch_x[0])  # convert each list to array
        batch_x[1] = np.array(batch_x[1])
        batch_y = np.array(batch_y)
        yield batch_x, batch_y

Then, make the model using functional tensorflow. When you fit the data, call your generator with the args needed:

history = model.fit(generator('train'),
                             validation_data = generator('validate'))