Getting an IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

Question

I am trying to implement a multiple input model in keras tensorflow with a custom generator as shown here Create a mixed data generator (images,csv) in keras in the accepted answer:

import random
import pandas as pd
import numpy as np
from glob import glob
from keras.preprocessing import image as krs_image

# Create the arguments for image preprocessing
data_gen_args = dict(
    horizontal_flip=True,
    brightness_range=[0.5, 1.5],
    shear_range=10,
    channel_shift_range=50,
    rescale=1. / 255,
)

# Create an empty data generator
datagen = ImageDataGenerator()

# Read the image list and csv
image_file_list = glob(f'{images_dir}/{split}/**/*.JPG', recursive=True)
df = pd.read_csv(f'{csv_dir}/{split}.csv', index_col=csv_data[0])
random.shuffle(image_file_list)

def custom_generator(images_list, dataframe, batch_size):
    i = 0
    while True:
        batch = {'images': [], 'csv': [], 'labels': []}
        for b in range(batch_size):
            if i == len(images_list):
                i = 0
                random.shuffle(images_list)
            # Read image from list and convert to array
            image_path = images_list[i]
            image_name = os.path.basename(image_path).replace('.JPG', '')
            image = krs_image.load_img(image_path, target_size=(img_height, img_width))
            image = datagen.apply_transform(image, data_gen_args)
            image = krs_image.img_to_array(image)

            # Read data from csv using the name of current image
            csv_row = dataframe.loc[image_name, :]
            label = csv_row['class']
            csv_features = csv_row.drop(labels='class')

            batch['images'].append(image)
            batch['csv'].append(csv_features)
            batch['labels'].append(label)

            i += 1

        batch['images'] = np.array(batch['images'])
        batch['csv'] = np.array(batch['csv'])
        # Convert labels to categorical values
        batch['labels'] = np.eye(num_classes)[batch['labels']]

        yield [batch['images'], batch['csv']], batch['labels']

However, I get the following index error. Any help is greatly appreciated.

ashraful16 · Answer 1 · 2020-09-16T11:18:17.883

1

I think you are trying to run block by block and try to running again previously executed block. Also, there is no problem in this code. Put the whole code in a single block and run again (or restart kernel and run all at once). If in this way your problem isn't solved, you can try my shared script. You can Copy and run my Colab file here, hope you can solve your problem.

edited Sep 16 '20 at 11:18

answered Sep 16 '20 at 11:04

ashraful16

2,742
3
11
32

Can you please clarify what you mean by putting the code in a single block? I am running it on Colab and I have get_model() function, custom generator and model_fit.generator( ) in separate cells. Did you mean to put everything in a single cell? Thank You for sharing your Colab file, but I am trying to use a custom generator. – Dushi Fdz Sep 16 '20 at 15:57
I tried implementing with all code together in the same cell, but the error remains. – Dushi Fdz Sep 16 '20 at 17:14
I think the easiest way you can do, copy my colab and just run. – ashraful16 Sep 16 '20 at 17:17
In your colab, there is no function for data generator. I cannot load all images into RAM. So, I am not sure how to adapt it in this case where I am trying to create a generator that uses both images and attributes and yield ([x1,x2],y). Isn't the problem within the custom generator and model_fit.generator here? Should those two be in a loop? – Dushi Fdz Sep 16 '20 at 17:31

score 0 · Accepted Answer · answered Sep 18 '20 at 01:20

0

The indexing error was due to this part:

batch['labels'] = np.eye(num_classes)[batch['labels']]

Instead of 'np.eye' I used keras 'to_catgorical' to convert the labels into one hot encoding and it worked.

answered Sep 18 '20 at 01:20

Dushi Fdz

161
3
22

Getting an IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

2 Answers2