17

For instance, after I have created my operations, fed the batch data through the operation and run the operation, does tf.train.batch automatically feed in another batch of data to the session?

I ask this because tf.train.batch has an attribute of allow_smaller_final_batch which makes it possible for the final batch to be loaded as a size lesser than the indicated batch size. Does this mean even without a loop, the next batch could be automatically fed? From the tutorial codes I am rather confused. When I load a single batch, I get literally a single batch size of shape [batch_size, height, width, num_channels], but the documentation says it Creates batches of tensors in tensors. Also, when I read the tutorial code in the tf-slim walkthrough tutorial, where there is a function called load_batch, there are only 3 tensors returned: images, images_raw, labels. Where are 'batches' of data as explained in the documentation?

Thank you for your help.

Soroush
  • 1,055
  • 2
  • 18
  • 26
kwotsin
  • 2,882
  • 9
  • 35
  • 62

3 Answers3

18

... does tf.train.batch automatically feeds in another batch of data to the session?

No. Nothing happens automatically. You must call sess.run(...) again to load a new batch.

Does this mean even without a loop, the next batch could be automatically fed?

No. tf.train.batch(..) will always load batch_size tensors. If you have for example 100 images and a batch_size=30 then you will have 3*30 batches as in you can call sess.run(batch) three times before the input queue will start from the beginning (or stop if epoch=1). This means that you miss out 100-3*30=10 samples from training. In case you do not want to miss them you can do tf.train.batch(..., allow_smaller_final_batch=True) so now you will have 3x 30-sample-batches and 1x 10-sample-batch before the input queue will restart.

Let me also elaborate with a code sample:

queue = tf.train.string_input_producer(filenames,
        num_epochs=1) # only iterate through all samples in dataset once

reader = tf.TFRecordReader() # or any reader you need
_, example = reader.read(queue)

image, label = your_conversion_fn(example)

# batch will now load up to 100 image-label-pairs on sess.run(...)
# most tf ops are tuned to work on batches
# this is faster and also gives better result on e.g. gradient calculation
batch = tf.train.batch([image, label], batch_size=100)

with tf.Session() as sess:
    # "boilerplate" code
    sess.run([
        tf.local_variables_initializer(),
        tf.global_variables_initializer(),
    ])
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess, coord=coord)

    try:
        # in most cases coord.should_stop() will return True
        # when there are no more samples to read
        # if num_epochs=0 then it will run for ever
        while not coord.should_stop():
            # will start reading, working data from input queue
            # and "fetch" the results of the computation graph
            # into raw_images and raw_labels
            raw_images, raw_labels = sess.run([images, labels])
    finally:
        coord.request_stop()
        coord.join(threads)
Samuel Bushi
  • 341
  • 3
  • 16
bodokaiser
  • 15,122
  • 22
  • 97
  • 140
  • Thank you for your answer. May I know if there is a for loop needed to continue loading the next batch? Are you currently loading 100 examples into one batch if there are only 100 examples in the training dataset? From most implementations I see in TF, there seem to be a loop but mine doesn't have one, so I'm unsure about this. Further, for the code below the boiler plate code, I used tf-slim's slim.learning.train instead, which I don't think has a loop except for the number of train steps taken. – kwotsin Jan 16 '17 at 12:34
  • Yes, if there are only 100 examples in the training dataset this would result in only one batch in my example. I cannot recommend `tf.contrib.slim` as it has only few documentation and is not maintained very well. It is also unclear if they will continue development at all. – bodokaiser Jan 16 '17 at 12:45
  • If there are more than 100 examples in the dataset, how would you suggest I load the next batch? If I were to simply call tf.train.batch again througha for loop, would it call the same examples? – kwotsin Jan 16 '17 at 12:47
  • Just as done in my code sample through `while not coord.should_stop()`. – bodokaiser Jan 16 '17 at 13:01
  • In your code, 100 examples from the entire training dataset are loaded at once but if I want a batch_size of 30 from the 100 examples, would the coord know? – kwotsin Jan 16 '17 at 13:08
  • if you pass `batch_size=30` to `tf.train.batch(..., batch_size=30)` yes it would. – bodokaiser Jan 16 '17 at 17:01
  • I apologize for the confusion; I meant if I I want to load using the method you written by training 100 examples in batches of 30,30,30,10, will the code `while not coord.should_stop()` be sufficient to read 100 examples in batches of 30 this way? Thank you. – kwotsin Jan 16 '17 at 17:04
  • In this case you need to use `tf.train.batch(..., batch_size=100, allow_smaller_final_batch=True)` but try it out yourself. – bodokaiser Jan 16 '17 at 17:24
  • 1
    Does it make sense to say `batch_image, batch_label = tf.train.batch([image, label], batch_size=100)` and inside tf.Session(), pull them out via `raw_images, raw_labels = sess.run([batch_image, batch_label])` in order to fully utilize the batch-processing? In the current code, it seems we are still pulling one instance at a time because we did not run on `batch`, right? – greeness Jun 13 '17 at 05:00
  • How does ´coord´ know that it has to handle ´batch´, if you do not pass it anywhere? Also ´images´ seems to be undefined. – Soerendip Dec 16 '17 at 21:24
0

You need to call sess.run and pass the batch to it everytime when you want to load the next batch. See the code below.

img = [0,1,2,3,4,5,6,7,8]
lbl = [0,1,2,3,4,5,6,7,8]
images = tf.convert_to_tensor(img)
labels = tf.convert_to_tensor(lbl)
input_queue = tf.train.slice_input_producer([images,labels])
sliced_img = input_queue[0]
sliced_lbl = input_queue[1]

img_batch, lbl_batch = tf.train.batch([sliced_img,sliced_lbl], batch_size=3)
with tf.Session() as sess:
    coord   = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)

    for i in range(0,3): #batch size
        image_batch,label_batch = sess.run([img_batch,lbl_batch ])
        print(image_batch, label_batch)

    coord.request_stop()
    coord.join(threads)

the answer would be something like this:

[4,1,8] [4,1,8]

[2,3,7] [2,3,7]

[2,6,8] [2,6,8]

Abdul Bari
  • 1
  • 1
0

I made the modification to the code from https://github.com/tensorflow/models/blob/master/research/slim/slim_walkthrough.ipynb and bodokaiser answer from the above post. Please note that this is from the evaluation scrip on https://github.com/tensorflow/models/tree/master/research/slim, eval_image_classifier.py. The most important modification to the eval_image_classifier.py code is to add num_epochs=1 to the DatasetDataProvider line. That way, all the images would be accessed once for inference.

provider = slim.dataset_data_provider.DatasetDataProvider(
    dataset,
    shuffle=False,
    common_queue_capacity=2 * FLAGS.batch_size,
    common_queue_min=FLAGS.batch_size, num_epochs=1)
[image, label] = provider.get(['image', 'label'])
images, labels = tf.train.batch(
    [image, label],
    batch_size=FLAGS.batch_size,
    num_threads=FLAGS.num_preprocessing_threads,
    capacity=1 * FLAGS.batch_size)
with tf.Session() as sess:
     sess.run([tf.local_variables_initializer(),
               tf.global_variables_initializer(),])
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess, coord=coord)
    try:
        while not coord.should_stop():
            np_image, np_label = sess.run([images, labels])
    except:
        coord.request_stop()
        coord.join(threads)
Madhumitha Sakthi
  • 11
  • 6