0

Note: I have already read How to reshape BatchDataset class tensor? but the solution doesn't apply here.

As mentioned in Train a neural network with input as sliding windows of a matrix with Tensorflow / Keras, and memory issues, I need to train a neural network with all sliding windows of shape (16, 2000) from a matrix of shape (10000, 2000). (In my other post it's 100k, but here 10k is ok).

import tensorflow
import numpy as np
X = np.array(range(20000000)).reshape(10000, 2000)  
X = tensorflow.keras.preprocessing.timeseries_dataset_from_array(X, None, 16, sequence_stride=1, sampling_rate=1, batch_size=32)
for b in X:
    print(b)  # Tensor of shape (32, 16, 2000)
    break

The problem is that I need to feed it into a model which requires a (..., 16, 2000, 1) shape.

model = Sequential()
model.add(Conv2D(16, kernel_size=(9, 9), activation='relu', input_shape=(16, 2000, 1), padding='same'))
...
model.fit(X, Y, epochs=8)

I tried

X = tensorflow.reshape(X, (-1, 16, 2000, 1))

without success.

How to do this, i.e. have the output of timeseries_dataset_from_array of shape (..., 16, 2000, 1)?

Basj
  • 41,386
  • 99
  • 383
  • 673

1 Answers1

1

To apply a transformation to each element of a tf.data.Dataset you should use the tf.data.Dataset.map function. In your case, you could define the function with a lambda, using tf.expand_dims:

ds = tensorflow.keras.preprocessing.timeseries_dataset_from_array(X, None, 16, sequence_stride=1, sampling_rate=1, batch_size=32)
ds_correct_shape = ds.map(lambda x: tf.expand_dims(x,axis=-1))

If we check the shape of the first element:

>>> for elem in ds_correct_shape.take(1):
        print(f"{elem.shape=}")
elem.shape=TensorShape([32, 16, 2000, 1])
Lescurel
  • 10,749
  • 16
  • 39
  • It should work out of the box. Without the Traceback, it's hard to debug though. – Lescurel Jun 23 '21 at 12:52
  • More precisely @Lescurel, I have `X = timeseries_dataset_from_array(X, None, 16, ...)` and `Y = timeseries_dataset_from_array(Y, None, 16, ...)`. Then `model.fit(X, Y)` does not work, one has to group X, Y inside a *single* dataset `ds` and then `model.fit(ds)`, how would you do that? (create `ds` from `X` and `Y`) – Basj Jun 23 '21 at 12:58
  • 1
    Ah, then, you can use `tf.data.Dataset.zip(X,Y)` to create a single dataset. – Lescurel Jun 23 '21 at 13:24