Keras: load images batch wise for large dataset

Question

Its is possible in keras to load only one batch in memory at a time as I have 40GB dataset of images.

If dataset is small I can used ImageDataGenerator to generator batches but due large dataset I can't load all images in memory.

Is there any method in keras to do something similar to following tensorflow code:

path_queue = tf.train.string_input_producer(input_paths, shuffle= False)
paths, contents = reader.read(path_queue)
inputs = decode(contents)
input_batch = tf.train.batch([inputs], batch_size=2)

I am using this method to serialize inputs in tensorflow but I don't know how to achieve this task in Keras.

what about [`train_on_batch`](https://keras.io/models/sequential/)? — desertnaut, Nov 09 '17 at 11:08

Daniel Möller · Accepted Answer · 2017-11-09T11:51:09.080

32

Keras has the method fit_generator() in its models. It accepts a python generator or a keras Sequence as input.

You can create a simple generator like this:

fileList = listOfFiles     

def imageLoader(files, batch_size):

    L = len(files)

    #this line is just to make the generator infinite, keras needs that    
    while True:

        batch_start = 0
        batch_end = batch_size

        while batch_start < L:
            limit = min(batch_end, L)
            X = someMethodToLoadImages(files[batch_start:limit])
            Y = someMethodToLoadTargets(files[batch_start:limit])

            yield (X,Y) #a tuple with two numpy arrays with batch_size samples     

            batch_start += batch_size   
            batch_end += batch_size

And fit like this:

model.fit_generator(imageLoader(fileList,batch_size),steps_per_epoch=..., epochs=..., ...)

Normally, you pass to steps_per_epoch the number of batches you will take from the generator.

You can also implement your own Keras Sequence. It's a little more work, but they recommend using this if you're going to make multi-thread processing.

edited Nov 09 '17 at 11:51

answered Nov 09 '17 at 11:31

Daniel Möller

84,878
18
192
214

the X of the tuple is an array of arrays since it is the list of images? – Helder Jun 04 '18 at 01:52
No, it must be a single array for keras to work. (This means a batch can only have all images of the same size). This is clear in the shapes required by convolution layers: `(batch_size, width,heigth,channels)`. – Daniel Möller Jun 04 '18 at 02:40
2

If you have a list of images all the same size you can `X = np.array(X)`. – Daniel Möller Jun 04 '18 at 03:45
so, if I have a batch of 50 images 256 x 256, is ok X be of 50 x 256 x 256? – Helder Jun 04 '18 at 13:22
2

It must be 4D: `(50, 256,256 ,3)`, for instance, for RGB images. Or `(50,256,256,1)` for grayscale images. – Daniel Möller Jun 04 '18 at 13:29
great, thank you! just one last question. In autoencoders, the yelded tuple must to be equal, I mean, the same set of input images must to be the output, right? – Helder Jun 04 '18 at 13:35
2

Yes, for traditional autoencoders you'd `yield (x,x)`. – Daniel Möller Jun 04 '18 at 13:55
if you want random shuffle, do you have to put it in the function or Keras does it for you? – Shantanu Shinde Sep 18 '20 at 00:33
@ShantanuShinde, It's impossible to shuffle this `yield` type of generator, you must take care of it yourself. You can try to create a `keras.utils.Sequence` type of generator, then Keras can shuffle "the batches", not the data. – Daniel Möller Sep 19 '20 at 04:49
@DanielMöller How would you add the validation training set? – henry May 15 '21 at 17:18
@henry, the same way you would before, or a generator too. – Daniel Möller May 16 '21 at 00:58
@DanielMöller Thanks a lot for your answer! So, I could simply re-use the same generator but load a validation dataset. – henry May 16 '21 at 10:38

Keras: load images batch wise for large dataset

1 Answers1

Linked