Data Augmentation Image Data Generator Keras Semantic Segmentation

Question

I'm fitting full convolutional network on some image data for semantic segmentation using Keras. However, I'm having some problems overfitting. I don't have that much data and I want to do data augmentation. However, as I want to do pixel-wise classification, I need any augmentations like flips, rotations, and shifts to apply to both feature images and the label images. Ideally I'd like to use the Keras ImageDataGenerator for on-the-fly transformations. However, as far as I can tell, you cannot do equivalent transformations on both the feature and label data.

Does anyone know if this is the case and if not, does anyone have any ideas? Otherwise, I'll use other tools to create a larger dataset and just feed it in all at once.

Thanks!

I had exactly the same problem, I also thought to ImageDataGenerator and I didn't find any solution. So I did it myself :-( — FiReTiTi, Jul 02 '16 at 06:35
Could I ask what tool you ended up using for this process to ensure that the same transformations were applied to both the features and the data? — TSW, Jul 02 '16 at 22:43
You can do this relatively easily by creating your own batch generator where you augment inputs/outputs the same way and then call `model.train_on_batch`. Make sure to shuffle the data yourself as this is normally taken care of by `model.fit`. — Mikael Rousson, Jul 03 '16 at 09:28
My own java code that I apply to the batch, or the batch + ground truth. — FiReTiTi, Jul 05 '16 at 08:11

score 18 · Answer 1 · answered Feb 25 '17 at 23:22

18

Yes you can. Here's an example from Keras's docs. You zip together two generators seeded with the same seeds and the fit_generator them. https://keras.io/preprocessing/image/

# we create two instances with the same arguments 
data_gen_args = dict(featurewise_center=True,
                     featurewise_std_normalization=True,
                     rotation_range=90.,
                     width_shift_range=0.1,
                     height_shift_range=0.1,
                     zoom_range=0.2) 
image_datagen = ImageDataGenerator(**data_gen_args) 
mask_datagen = ImageDataGenerator(**data_gen_args)

# Provide the same seed and keyword arguments to the fit and flow methods seed = 1 
image_datagen.fit(images, augment=True, seed=seed) 
mask_datagen.fit(masks, augment=True, seed=seed)

image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
    'data/masks',
    class_mode=None,
    seed=seed)

# combine generators into one which yields image and masks 
train_generator = zip(image_generator, mask_generator)

model.fit_generator(
    train_generator,
    samples_per_epoch=2000,
    nb_epoch=50)

answered Feb 25 '17 at 23:22

Dennis Sakva

1,447
2
13
26

1

i get stuck when i try to zip them together, do you have an idea? `train_generator = zip(image_generator, mask_generator)`either memory limit or never ending execution – olix20 Apr 01 '17 at 15:58
2

use itertools.izip() instead, that gives you the generator version of zip() – dgorissen Jul 13 '17 at 08:59
@Dennis I've a `onehot mask encoding` function and would like to convert all the masks to this encoding. How/where I can call my function in terms of the `mask_generator`? – Tin Jul 29 '18 at 13:47
1

flow_from_directory() requires subdirectory represent each class. Only when your data format in this way . Or you have to format before using it. – Tina Liu Apr 29 '19 at 03:12
I can't figure out what `images` and `masks` should be. I know they represent input data `X` and target data `Y` but I thought `X` and `Y` are getting pulled from the generators.. Hmm. – Yeow_Meng Feb 28 '20 at 20:47
I think this is exclusively for training datasets. You still need to include the data for validation, which will demand more two blocks of flow_from_directory(...). I've tried ImageDataGenerator(..., validation_split=0.15, ...) (see [here](https://stackoverflow.com/questions/55963087/in-keras-imagedatagenerator-is-validation-split-parameter-a-kind-of-k-fold)) among the params, but got 99% of accuracy. I don't know, did not feel good to use validation_split, then, splited the data by myself and added two more blocks as mentioned. – 66lotte Oct 19 '20 at 23:22

score 17 · Accepted Answer · answered Aug 21 '16 at 08:29

There are works on extending ImageDataGenerator to be more flexible for exactly these type of cases (see in this issue on Github for examples).

Additionally, as mentioned by Mikael Rousson in the comments, you can easily create your own version of ImageDataGenerator yourself, while leveraging many of its built-in functions to make it easier. Here is an example code I've used for an image denoising problem, where I use random crops + additive noise to generate clean and noisy image pairs on the fly. You could easily modify this to add other types of augmentations. After which, you can use Model.fit_generator to train using these methods.

from keras.preprocessing.image import load_img, img_to_array, list_pictures

def random_crop(image, crop_size):
    height, width = image.shape[1:]
    dy, dx = crop_size
    if width < dx or height < dy:
        return None
    x = np.random.randint(0, width - dx + 1)
    y = np.random.randint(0, height - dy + 1)
    return image[:, y:(y+dy), x:(x+dx)]

def image_generator(list_of_files, crop_size, to_grayscale=True, scale=1, shift=0):
    while True:
        filename = np.random.choice(list_of_files)
        try:
            img = img_to_array(load_img(filename, to_grayscale))
        except:
            return
        cropped_img = random_crop(img, crop_size)
        if cropped_img is None:
            continue
        yield scale * cropped_img - shift
def corrupted_training_pair(images, sigma):
    for img in images:
        target = img
        if sigma > 0:
            source = img + np.random.normal(0, sigma, img.shape)/255.0
        else:
            source = img
        yield (source, target)
def group_by_batch(dataset, batch_size):
    while True:
        try:
            sources, targets = zip(*[next(dataset) for i in xrange(batch_size)])
            batch = (np.stack(sources), np.stack(targets))
            yield batch
        except:
            return
def load_dataset(directory, crop_size, sigma, batch_size):
    files = list_pictures(directory)
    generator = image_generator(files, crop_size, scale=1/255.0, shift=0.5)
    generator = corrupted_training_pair(generator, sigma)
    generator = group_by_batch(generator, batch_size)
    return generator

You can then use the above like so:

train_set = load_dataset('images/train', (patch_height, patch_width), noise_sigma, batch_size)
val_set = load_dataset('images/val', (patch_height, patch_width), noise_sigma, batch_size)
model.fit_generator(train_set, samples_per_epoch=batch_size * 1000, nb_epoch=nb_epoch, validation_data=val_set, nb_val_samples=1000)

What does the flow function do while using data generators in Keras ? — tauseef_CuriousGuy, Oct 11 '16 at 16:32
The flow function is for creating a data generator from directory of files or numpy arrays. However, this have nothing with this question, or my answer. I suggest you open a new question with your exact problem and not as a comment on an answer to an unrelated question. — Or Sharir, Oct 11 '16 at 17:26

Data Augmentation Image Data Generator Keras Semantic Segmentation

2 Answers2

Linked