0

I have installed Mongodb running inside a Kubernetes cluster with 2 mongodb replicaset and 1 arbiter within each replicaset (mongodb-0, mongodb-1) I have enabled mongodb-exporter

my-feature            mongodb-0                                                         2/2     Running   0               41h
my-feature            mongodb-1                                                         2/2     Running   0               41h
my-feature            mongodb-arbiter-0                                                 1/1     Running   0               41h

When I tried to restore database from a snapshot (AWS EC2 volume). mongodb-0, mongodb-arbiter pod can start normally, but mongodb-1 can not start and got "crashloopbackoff" When I inspect the log, I saw the mongodb-metrics-exporter show this error log

Error log:

Stream closed EOF for my-feature/mongodb-1 (mongodb)
metrics time="2023-08-28T05:23:15Z" level=error msg="Cannot connect to MongoDB: cannot connect to MongoDB: server selection error: context canceled, current topology: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp [::1]:27017: connect: connection refused }, ] }"
metrics 2023/08/28 05:23:15 http: panic serving 10.0.2.198:44132: runtime error: invalid memory address or nil pointer dereference
metrics goroutine 97 [running]:
metrics net/http.(*conn).serve.func1()
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:1802 +0xb9
metrics panic({0xb36880, 0x1264f80})
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/runtime/panic.go:1047 +0x266
metrics go.mongodb.org/mongo-driver/mongo.newDatabase(0x0, {0xbf9d4a, 0x5}, {0x0, 0x30, 0xc00029e400})
metrics     /home/runner/go/pkg/mod/go.mongodb.org/mongo-driver@v1.9.1/mongo/database.go:47 +0x5c
metrics go.mongodb.org/mongo-driver/mongo.(*Client).Database(...)
metrics     /home/runner/go/pkg/mod/go.mongodb.org/mongo-driver@v1.9.1/mongo/client.go:837
metrics github.com/percona/mongodb_exporter/exporter.getClusterRole({0xd97d80, 0xc00009a660}, 0x40d1f4)
metrics     /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/topology_info.go:170 +0x8b
metrics github.com/percona/mongodb_exporter/exporter.(*topologyInfo).loadLabels(0xc000099560, {0xd97d80, 0xc00009a660})
metrics     /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/topology_info.go:103 +0xeb
metrics github.com/percona/mongodb_exporter/exporter.newTopologyInfo({0xd97d80, 0xc00009a660}, 0x0)
metrics     /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/topology_info.go:73 +0x8b
metrics github.com/percona/mongodb_exporter/exporter.(*Exporter).Handler.func1({0xd95020, 0xc00048a000}, 0xc000486000)
metrics     /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/exporter.go:304 +0x3a5
metrics net/http.HandlerFunc.ServeHTTP(0x0, {0xd95020, 0xc00048a000}, 0x0)
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:2047 +0x2f
metrics net/http.(*ServeMux).ServeHTTP(0x0, {0xd95020, 0xc00048a000}, 0xc000486000)
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:2425 +0x149
metrics net/http.serverHandler.ServeHTTP({0xc000098b10}, {0xd95020, 0xc00048a000}, 0xc000486000)
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:2879 +0x43b
metrics net/http.(*conn).serve(0xc0000ba1e0, {0xd97db8, 0xc000346c60})
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:1930 +0xb08
metrics created by net/http.(*Server).Serve
metrics     /opt/hostedtoolcache/go/1.17.11/x64/src/net/http/server.go:3034 +0x4e8

Do you have any suggestion on this error, thanks!

The lastime, we did the restore process, it worked fine, but this time we got this error

PikaLong
  • 1
  • 2
    I can only guess, but based on _`[::1]:27017: connect: connection refused`_, I suspect `localhost` is now being translated to an IPv6 address. Perhaps if you specified `127.0.0.1` instead of `localhost`, it might work? Alternatively, you could enable IPv6 for your MongoDB servers in their config files. – rickhg12hs Aug 30 '23 at 02:51
  • See https://stackoverflow.com/questions/74609210/cant-connect-to-mongodb-6-0-server-locally-using-node-js-driver/74610881#74610881 – Wernfried Domscheit Aug 30 '23 at 05:06

1 Answers1

1

Your replicaSet best to be initialized via the hostnames not via IP-addresses since this can easily change in kubernetes , and also you may need to update this setting in the mongodb config that need to point where the instance will listen:

net:
  bindIp: {{ $$hostname }}
R2D2
  • 9,410
  • 2
  • 12
  • 28