Questions tagged [tf.data.dataset]
145 questions
13
votes
1 answer
How exactly does tf.data.Dataset.interleave() differ from map() and flat_map()?
My current understanding is:
Different map_func: Both interleave and flat_map expect "A function mapping a dataset element to a dataset". In contrast, map expects "A function mapping a dataset element to another dataset element".
Arguments: Both…
gebbissimo
- 2,137
- 2
- 25
- 35
6
votes
2 answers
Tensorflow dataset with multiple inputs and target
I am trying to implement a model with the ArcFace Layer:
https://github.com/4uiiurz1/keras-arcface
to this extend I created a tf.data.dataset like so:
images= tf.data.Dataset.from_tensor_slices(train.A_image.to_numpy())
target =…
Olli
- 906
- 10
- 25
5
votes
1 answer
How to use tf.data in tensorflow to read .csv files?
I have three different .csv datasets that I typically read using pandas and train deep learning models with. Each data is a n by m matrix where n is the number of samples and m is the number of features. After reading the data, I do some reshaping…
khemedi
- 774
- 3
- 9
- 19
5
votes
1 answer
how to access tf.data.Dataset within a keras custom callback?
I have written a custom keras callback to check the augmented data from a generator. (See this answer for the full code.) However, when I tried to use the same callback for a tf.data.Dataset, it gave me an error:
File…
craq
- 1,441
- 2
- 20
- 39
5
votes
0 answers
Tensorflow 2 - AttributeError: '_NestedVariant' object has no attribute 'batch'
In Chapter 17 of the book "Hands on machine learning with scikit-learn and tensorflow 2.0", they split a sequential dataset into multiple windows by using tf.data.Dataset and the window() method:
n_steps = 100
window_length = n_steps + 1 # target =…
ebeninki
- 909
- 1
- 12
- 34
4
votes
1 answer
How can I remove or omit data using map method for tf.data.Dataset objects?
I am using tensorflow 2.3.0
I have a python data generator-
import tensorflow as tf
import numpy as np
vocab = [1,2,3,4,5]
def create_generator():
'generates a random number from 0 to len(vocab)-1'
count = 0
while count < 4:
x…
n0obcoder
- 649
- 8
- 24
3
votes
1 answer
Tensorflow: how to feed a variable-time-step input to a RNN
I have a simple X_train and Y_train data:
x_train = [
array([ 6, 1, 9, 10, 7, 7, 1, 9, 10, 3, 10, 1, 4]),
array([ 2, 8, 8, 1, 1, 4, 2, 5, 1, 2, 7, 2, 1, 1, 4, 5, 10, 4])
]
y_train = [23, 17]
Arrays are numpy arrays.
I am…
Fab
- 142
- 8
3
votes
1 answer
TensorFlow 2.6: num_parallel_calls is greater than 1 but only one CPU core is used most of the time
I wrote a TF data pipeline that looks something like this (TF 2.6):
def parse(img):
image = tf.image.decode_png(img, channels=3)
image = tf.reshape(image, IMG_SHAPE)
image = tf.cast(image, TARGET_DTYPE)
return image
def…
Daniil Novikov
- 61
- 4
3
votes
1 answer
Normalize tf.data.Dataset
I have a tf.data.Dataset of images with input shape (batch-size, 128, 128, 2) and target shape (batch-size, 128, 128, 1) where the inputs are 2-channel images (complex-valued images with two channels representing real and imaginary part) and the…
psj
- 356
- 3
- 18
3
votes
1 answer
Specifying class or sample weights in Keras for one-hot encoded labels in a TF Dataset
I am trying to train an image classifier on an unbalanced training set. In order to cope with the class imbalance, I want either to weight the classes or the individual samples. Weighting the classes does not seem to work. And somehow for my setup I…
user1158795
- 81
- 1
- 1
- 5
3
votes
1 answer
How does Model.fit() method's shuffle deals with Batches when using a tf.data.Dataset?
I am using tensorflow 2.
When using the Model.fit() method with a tf.data.Dataset, the argument 'batch_size' is ignored. Thus to train my model on batches, I have to first change my dataset of samples into a dataset of batches of samples by calling…
Matt
- 31
- 3
3
votes
1 answer
How to use tf.data.Dataset with kedro?
I am using tf.data.Dataset to prepare a streaming dataset which is used to train a tf.kears model. With kedro, is there a way to create a node and return the created tf.data.Dataset to use it in the next training node?
The MemoryDataset will…
evolved
- 1,850
- 19
- 40
3
votes
1 answer
How to access Tensor shape within .map function?
I have a dataset of audios in multiple lengths, and I want to crop all of them in 5 second windows (which means 240000 elements with 48000 sample rate). So, after loading the .tfrecord, I'm doing:
audio, sr = tf.audio.decode_wav(image_data)
which…
Leonardo
- 155
- 2
- 10
2
votes
1 answer
`py_function` causes `ragged_batch()` not working in `tf.data`
I'm working on an object detection project, and use tf.data.Dataset input pipeline to load local data. Because object detection requires not only image but also annotations, and the different dimension of annotations makes it even harder. I tried…
Yiming Designer
- 423
- 3
- 10
2
votes
2 answers
Issue in tf.data pipeline for TensorFlow model
I have an application in which I need to setup a pipeline using tf.data. The data I have is stored in .mat files created in Matlab and contains three variables "s_matrix" which is a 224x224x3 double array, a "frame" which is 1024x1 complex double…
malik12
- 183
- 1
- 13