4

Keras's RepeatVector layer allows me to repeat a given vector/Tensor n times: https://www.tensorflow.org/api_docs/python/tf/keras/layers/RepeatVector.

However, it appears n has to be a constant. Is there any way to specify this parameter dynamically?

Use case: I'm generating variable length sequences, and RepeatVector will be followed by an LSTM layer.

flow2k
  • 3,999
  • 40
  • 55
  • https://stackoverflow.com/questions/62166820/how-to-use-timedistributed-layer-for-predicting-sequences-of-dynamic-length-pyt/62167747#62167747 – Marco Cerliani Jun 04 '20 at 18:08

2 Answers2

2

It can be a symbolic tensor. Therefore, you can use the backend function shape() (or alternatively tf.shape()) to dynamically find the number of timesteps from input tensor of LSTM layer:

from keras import backend as K

reps = RepeatVector(K.shape(lstm_input)[1])(lstm_out)
today
  • 32,602
  • 8
  • 95
  • 115
  • Great to hear it can be a Tensor, although in my case, `shape()` is still a constant, since I'm using padding and `Masking` in the input RNN layer. I'm thinking of using a Lambda layer to compute `n` dynamically - what do you think of this approach? – flow2k Aug 21 '19 at 18:09
  • 1
    @flow2k As long as the following layers support dynamic-length inputs, you can do that. – today Aug 21 '19 at 18:11
  • Also, sidebar question: are you aware of a way to check or debug the value of a Tensor, like `K.shape()` or a `Lambda` in this case? I could use the wrong index in `shape` or have a bug in the Lambda, and I would never know (because the following LSTM can accept arbitrary length of input)... – flow2k Aug 21 '19 at 18:12
  • Great on your last point - an LSTM layer immediately follows which accepts dynamically-varying input, and it doesn't return a sequence, just the last output, so there shouldn't be a problem. – flow2k Aug 21 '19 at 18:15
  • 1
    @flow2k You can use `tf.print()` to print the values of Tensors. – today Aug 21 '19 at 18:15
  • Just noticed - did you mean to write `(lstm_input)` in the last parenthesis? That's the input to the RepeatVector layer, as I understand. – flow2k Aug 21 '19 at 21:30
  • 1
    @flow2k I thought you have an LSTM layer which you get its last output, then repeat it, and then pass it to the next LSTM layer (something like in seq2seq). If that's not the case, then you are right: the input to `RepeatVector` is the one you want to repeat. – today Aug 22 '19 at 07:03
  • @today Can you please take a look into this question? https://stackoverflow.com/q/58900947/5904928 I am struggling for hours, couldn't find an answer. – Aaditya Ura Nov 17 '19 at 13:43
  • @today, how do you know if `RepeatVector` can take symbolic tensor because as per docs, it says `n` is an integer. I was trying to repeat a constant value via `repeat_elements` and `tile` based on the shape of `y_true` in the keras loss function but both of them expect an integer repetition factor which keras raises compile time error. But your solution worked. – CKM Dec 08 '19 at 17:15
0

For me, that code did not result in a model that would compile successfully. It complained that the tensor provided was not the output of a layer. The code that worked for me was:

sentence_length = Lambda(
    lambda t: tensorflow.shape(t)[1],
    name="sentence_length"
)(word_digests)

# sentence vector, repeated so available for each word
sentence_repeated = Lambda(
    lambda t: tensorflow.tile( tensorflow.expand_dims(t, 1), (1,sentence_length,1) ),
    name="sentence_repeated"
)(sentence_digest)
chrishmorris
  • 287
  • 1
  • 6