0

I'm currently struggling when using Capsule Network (Keras version: CapsNet).

Each time I run more than 2-5 predictions in a row (in side a loop) the results vary a lot. I have tried to change so mange things. I have changed the optimizer from ADAM to SGD as well - but I simply can't make it 100 % stable and thereby be able to reproduce a given run - once again.

How can I make CapsNet 100% reproducible every run?

PabloDK
  • 2,181
  • 2
  • 19
  • 42
  • Welcome to SO! When you place a question try to add a minimum content: input sample, expected output sample, what did you try, research and where are you stuck. What did you try? Show us... – David García Bodego Oct 24 '19 at 04:36

1 Answers1

1

The answer to this is long and involved. There's a blog post that goes into much more detail than I can here, but I'll try to capture the high level points.

  1. Set the PYTHONHASHSEED environment variable to 0 before running your python program.
  2. If you're running calculations on the GPU, that can result in non-reproducible results due to float-rounding. You can disable it and run all operations on the CPU by setting the CUDA_VISIBLE_DEVICES environment variable to an empty string in the same manner as before.
CUDA_VISIBLE_DEVICES="" PYTHONHASHSEED=0 python your_program.py
George Stocker
  • 57,289
  • 29
  • 176
  • 237
ad2004
  • 809
  • 6
  • 7
  • Thank you my friend. Unfortunately I have read and tried this out already and it didn't change anything. – PabloDK Oct 23 '19 at 21:13
  • @PabloDK In that case, it might be useful to convince yourself that you can actually obtain reproducible results within your environment using a very simple network like one of the Keras "getting started" examples (https://keras.io/getting-started/sequential-model-guide/). That might suggest possible directions to explore for your Capsule Network trouble. I hope this helps. – ad2004 Oct 24 '19 at 04:39
  • I have tried this as well and when running on the CPU it produces the same results but only in conjunction with the SGD optimizer. If I change the optimizer to ADAM or shift from CPU to GPU then the results varies a lot again :-( Is this really the reality and the standard when playing around with stuff like this? It seems to me like such a basic issue (something which all developers would run in to) - so again, it seems crazy no better options exists (for making gpu + adam possible in regards to 100% reproducible)? – PabloDK Oct 24 '19 at 10:50
  • Another suggestion is to use Keras with Tensorflow 2.0 in Google Colab. The following question has some example code in the answer section that seems to behave well (at least with a CPU): https://stackoverflow.com/a/58310632/7024006 Again, it might be useful to start with a simple network, substitute SGD and then Adam etc. to validate the environment. At least if there is some issue with Keras/Tensorflow, it will be with a more recent implementation. I hope this helps. – ad2004 Oct 24 '19 at 16:25