Using tf.data.Dataset.from_generator() with a function (tensorflow or numpy) as the generating source (instead of a file)

Question

The script is attempting to use a function (np.sin() in this case) as a generator designed to ultimately be pipelined into model for training (hence the tf.Session()). Unfortunately, I keep getting the error:

"ValueError: generator yielded an element of shape () where an element of shape (1,) was expected."

I've used generators in the past to pull data from .hdf5 files, but what I'm trying to do here is instead generate waveform data from a function within the callable class.

Calling the generating function outside of the tf.data.Dataset.from_generator() works as desired:

next(sine_wave_source())

import numpy as np
import tensorflow as tf

class sine_wave_source:

    def __init__(self,frequency = 1,sampling_frequency = 100):
        self.fc = frequency
        self.Fs = sampling_frequency
        self.time_vector = np.arange(0,1,1/self.Fs,dtype = 'float32')

    def __call__(self):
        for t in self.time_vector:
            yield np.sin(2*np.pi*self.fc*t,dtype = 'float32')

data_gen = tf.data.Dataset.from_generator(
            sine_wave_source(),
            output_types = (tf.float32),
            output_shapes = (tf.TensorShape([1])))

data_iterator = data_gen.make_initializable_iterator()

next_sample = data_iterator.get_next()

with tf.Session() as sess:
        sess.run(data_iterator.initializer)
        for ii in range(0,100):
           sample = sess.run([next_sample])
           print(sample)

score 3 · Accepted Answer · answered May 22 '19 at 14:59

3

With output_shapes = (tf.TensorShape([1])) you are indicating that each item in the dataset will be a one-dimensional tensor with one element. However, the generator in sine_wave_source returns scalar values. TensorFlow is strict here, it could just broadcast to the one-item vector, but considers it an error. You can either change output_shapes like:

data_gen = tf.data.Dataset.from_generator(sine_wave_source(),
                                          output_types=tf.float32,
                                          output_shapes=tf.TensorShape([]))

Or change the generator to return a list or an array, e.g.:

def __call__(self):
    for t in self.time_vector:
        yield [np.sin(2 * np.pi * self.fc * t, dtype='float32')]

answered May 22 '19 at 14:59

jdehesa

58,456
7
77
121

Thank you for the fast response! Your first suggestion worked marvelously, and not specifying a shape also tends to do the trick. Unfortunately I'm still getting the same error with your second suggestion of placing the yielded expression in brackets. I see your point with Tensorflow requiring tensors versus scalars, and intuitively I can't see why your solution would not work. – Nasty May 22 '19 at 15:09
@Nasty That's strange, I just tested it (TF 1.12, CPU and GPU, and TF 1.13) and it seems to work for me (I took the code you posted and just added the brackets). – jdehesa May 22 '19 at 15:21
Interesting, I'll continue to investigate, but I have at least one workaround for now. I'm using TF 1.13.1 GPU and Python 3.6. Thanks again for the help! – Nasty May 22 '19 at 15:25
Confirming both approaches, I had a different issue. Thanks again! – Nasty May 23 '19 at 23:24

Using tf.data.Dataset.from_generator() with a function (tensorflow or numpy) as the generating source (instead of a file)

1 Answers1

Linked