2

I have two computers: Ubuntu1 and Ubuntu2. Ubuntu1 runs MongoDB with database Sacred3. I want to connect from U2 to U1 via ssh and store there my experiment results.

What I tried and failed: 1. I installed mongo DB, created sacred3, I have ssh key to it. I edited /etc/mongod.conf adding:

# network interfaces net: port: 27017 bindIp: 0.0.0.0

Then I enabled port forwarding with

ssh -fN -i ~/.ssh/sacred_key-pair.pem -L 6666:localhost:27017 ubuntu@106.969.696.969 // (with proper ip)

so, as I undertstand, if I connect to my localhost:6666 it will be forwarded to 106.969.696.969:27017

So after that, I'm runnig an experiment with Sacred framework:

python exp1.py -m localhost:6666:sacred3

and this should write experiment to remote DB, HOWEVER i I get:

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

which is driving me mad. please help!

#

below contents of exp1.py:

from sacred import Experiment
from sacred.observers import MongoObserver

ex = Experiment()
ex.observers.append(MongoObserver.create())

def compute():
    summ = layer1 - layer2
    return summ


@ex.config
def my_config():

    hp_list = [{"neurons" : [32,32] , "dropout": 1.0},
            {"neurons" : [32,32] , "dropout": 0.7},
            {"neurons" : [32,16] , "dropout": 0.9},
            {"neurons" : [24,16] , "dropout": 0.9},
            {"neurons" : [24,8] , "dropout":  0.9},
            {"neurons" : [16,8] , "dropout":  0.9},
            {"neurons" : [64,64] , "dropout": 0.9},
            {"neurons" : [64,64] , "dropout": 0.7},
            {"neurons" : [64,32] , "dropout": 0.9},
            {"neurons" : [64,32] , "dropout": 0.7},
            {"neurons" : [48,32] , "dropout": 0.9},
            {"neurons" : [48,32] , "dropout": 0.7},
            {"neurons" : [48,16] , "dropout": 0.9},
            {"neurons" : [48,16] , "dropout": 0.7},]

    n_epochs = 2 


@ex.capture
def training_loop(hp_list, n_epochs):
    for j in hp_list:
        print("Epoch: ", n_epochs)
#       layer1 = random.randint(18,68)
#       layer2 = random.randint(18,68)
#       layer3 = random.randint(18,68)
        layer1 = j["neurons"][0]
        layer2 = j["neurons"][1]
        dropout_ratio = j["dropout"]


        print("WHATS UUUUUP",j, layer1, layer2, dropout_ratio, sep="_")
        # vae_training_loop_NN_DO(i, layer1, layer2, dropout_ratio )


@ex.automain
def my_main():
    training_loop()
MikolajM
  • 300
  • 1
  • 13
  • Looks like you're not passing the port to the script correctly and therefore connecting to localhost:27017 instead of localhost:6666 – r.delic Oct 15 '19 at 13:35
  • what do you mean? I run script with python exp1.py -m localhost:6666:sacred3 – MikolajM Oct 15 '19 at 13:46
  • afaik the `-m` switch [isn't for passing arguments](https://stackoverflow.com/questions/7610001/what-is-the-purpose-of-the-m-switch) to a script.. Can we see the contents of your script? – r.delic Oct 15 '19 at 14:03
  • I added contents of exp1.py just like it is here https://sacred.readthedocs.io/en/stable/observers.html – MikolajM Oct 15 '19 at 14:17

1 Answers1

1

According to the documentation supplied, it looks like you're creating two observers, or overriding the connection argument you passed with -m, with the MongoObserver.create()specified in the code which uses the default mongo host and port localhost:27017. You either supply the observer connection via the -m argument or in code, not both.

Try removing the MongoObserver.create() line altogether, or hardcoding the connection arguments: MongoObserver(url='localhost:6666', db_name='sacred3')

Also, it looks like your mongo host is not liking the binding to localhost so you should also replace localhost in your ssh command with 127.0.0.1 or [::1], e.g ssh -fN -i ~/.ssh/sacred_key-pair.pem -L 6666:127.0.0.1:27017 ubuntu@106.969.696.969 or ssh -fN -i ~/.ssh/sacred_key-pair.pem -L 6666:[::1]:27017 ubuntu@106.969.696.969

r.delic
  • 795
  • 7
  • 18
  • I removed line ex.observers.append(MongoObserver.create()) and now I'm getting this error: channel 2: open failed: connect failed: Connection refused – MikolajM Oct 15 '19 at 14:29
  • 1
    Now it looks like your mongo host is [not liking the binding to localhost](https://serverfault.com/questions/489192/ssh-tunnel-refusing-connections-with-channel-2-open-failed). Can you try replacing `localhost` in your ssh command with `127.0.0.1` or `[::1]`, e.g `ssh -fN -i ~/.ssh/sacred_key-pair.pem -L 6666:127.0.0.1:27017 ubuntu@106.969.696.969` – r.delic Oct 15 '19 at 14:42
  • 1
    Halleluyah, now it worked! Please add this comment to your answer so other people can see it all. – MikolajM Oct 16 '19 at 08:52
  • 1
    Done. Then you could also update your question with the second error so it shows up in search results for that error. – r.delic Oct 16 '19 at 09:36