0

I write a simple program to try out reading data in batch functions in TensorFlow but encountered a problem:

I created 6 simple csv files; each file contains 3 records like:

1.0,1.0,1.0,1.0,1
1.1,1.1,1.1,1.1,1
1.2,1.2,1.2,1.2,1

(the first 4 columns are feature and the fifth column is the label.) so totally 6 files have 6*3=18 records.

I try to read the files into 3 batches of 6 records/batch using reader, batch or shuffle_batch. When I don't specify num_epochs in string_input_producer the code works fine. But when I specify num_epochs the batch or shuffle_batch always throws OutOfRange error. The current_size is always zero...

Here is the code:

import tensorflow as tf
import os

csvFiles = os.listdir('./data')
csvFiles = [i for i in csvFiles if i[-4:]=='.csv' ]
csvFiles = ['./data/'+i for i in csvFiles]

print(csvFiles)

fileQ = tf.train.string_input_producer(csvFiles,shuffle=False,num_epochs=3)
reader = tf.TextLineReader()
key,value = reader.read(fileQ)
record_defaults = [[0.0], [0.0], [0.0], [0.0], [0]]
col1, col2, col3, col4, label = tf.decode_csv(value, record_defaults=record_defaults)
feature = tf.stack([col1, col2, col3, col4])
feature_batch, label_batch = tf.train.shuffle_batch([feature, label], batch_size=6, capacity=100, min_after_dequeue=1) # num_threads=3,

with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess,coord=coord)

    try:
        for i in range(3):
            featureBatch, labelBatch = sess.run([feature_batch, label_batch])
            print(featureBatch)
            print(labelBatch)
    except tf.errors.OutOfRangeError:    
        print("Done reading!")
    finally:
        coord.request_stop()

coord.join(threads)
print("**END**")

the output of OutOfRange error is here

please note the error was throwed when shuffle_batch was first called. I think it means not a single record could be read.

and even I changed the code to just read one record it throwed the same error: l,f=sess.run([label,feature])

This is a very simple code. Wonder what's wrong with it? Thank you very much!

pfm
  • 6,210
  • 4
  • 39
  • 44
Janet Wu
  • 1
  • and for the code I have another question about coord. I saw some sample code don't specify coord parm in: tf.train.start_queue_runners(sess=sess) but if I remove coord statements my program will go into dead loop. I wonder how the other sample code works without specifying coord?? – Janet Wu Jul 20 '17 at 03:53
  • Duplicate of : https://stackoverflow.com/questions/44874244/outofrangeerror-for-tf-train-string-input-producer/44875392#44875392 – Vijay Mariappan Jul 20 '17 at 04:30
  • Hi Vijay thanks for your input! However I don't think mine is duplicate to that question. My problematic code does use "coord". And if I don't specify num_pochs it works fine. – Janet Wu Jul 20 '17 at 06:05

2 Answers2

0

This is explained in the string doc of the method:

num_epochs: An integer (optional). If specified, string_input_producer produces each string from string_tensor num_epochs times before generating an OutOfRange error. If not specified, string_input_producer can cycle through the strings in string_tensor an unlimited number of times.

The OutOfRangeerror basically reproduces the StopIteration error Python raises when iterating on a list. See for example this answer.

pfm
  • 6,210
  • 4
  • 39
  • 44
0

After reading the other sample code, I found we need add: tf.local_variables_initializer().run() to initialize the variables. (Even I don't know why num_Epochs=3 needs be initialized.)

Now the code can work.

Janet Wu
  • 1