0

Using the keras simpleRNN layer, I am hitting this wall. I have two other models, one only with fully connected Dense layers, and one using LSTM which work as expected, so I don't think it's the data processing that is the issue.

For context, I am using the tf.keras reuters dataset, which comes tokenized, and the output data consists of 46 possible tags which I have categorized.

What the data looks like and how it's processed What the data looks like and how it's processed

Below is the model code.

modelRNN = Sequential()

modelRNN.add(Embedding(input_dim=maxFeatures, output_dim=256,input_shape=(maxWords,)))
modelRNN.add(SimpleRNN(1024))
#modelRNN.add(Activation("sigmoid"))
modelRNN.add(Dropout(0.8))

modelRNN.add(Dense(128))

modelRNN.add(Dense(46, activation="softmax"))


modelRNN.compile(
  optimizer='adam',
  loss='categorical_crossentropy',
  metrics=['accuracy'],
)

And I am fitting using the following parameters

historyRNN = modelRNN.fit(x_train, y_train,
  epochs=100,
  batch_size=512,
  shuffle=True,
  validation_data = (x_test,y_test)
  )

Fitting this model, consistently has a val_accuracy of 0,3762, and a val_loss of ~3,4. This "ceiling" can be clearly seen in the graph:

Graphs

Things I've tried: changing super parameters, changing the input data shape, trying different optimizers.

Any tip is appreciated, thank you. And thank you to the people that helped edit my posts to be more understandable :)

The graphs for the other two models, working on the same data:

Dense layers only

Dense graph

LSTM

LSTM graph

Stefan Horvath
  • 27
  • 6
  • 1
    Try changing the Dropout rate to something reasonable... i.e. between 0.1 and 0.2. .8 means that you are dropping 80% of the values at random out of that layer. – Pedro Marques Dec 08 '20 at 22:31
  • I tried that, and it didn't change anything. Another thing I've tried is add more layers but again nothing. However, changing the dimensions of the input data (max words and max features, I am changing the "ceiling" value. I'll try to see what I can do and post the answer. – Stefan Horvath Dec 09 '20 at 14:02
  • 1
    @StefanHorvath Found a similar issue [here](https://stackoverflow.com/questions/60319768/cant-import-tensorflow-in-python-windows-10-64-bit/61656869#61656869), it might help you. Thanks! –  Mar 22 '22 at 07:58

0 Answers0