UncategorizedMongoDbException: Command failed with error 10107 (NotMaster): 'not master' on server

Question

I am getting the following exception while executing my integration test cases for transactional feature of MongoDB.

org.springframework.data.mongodb.UncategorizedMongoDbException: Command failed with error 10107 (NotMaster): 'not master' on server localhost:60876. The full response is {"operationTime": {"$timestamp": {"t": 1615104939, "i": 1}}, "ok": 0.0, "errmsg": "not master", "code": 10107, "codeName": "NotMaster", "$gleStats": {"lastOpTime": {"$timestamp": {"t": 1615104939, "i": 1}}, "electionId": {"$oid": "7fffffff0000000000000001"}}, "lastCommittedOpTime": {"$timestamp": {"t": 0, "i": 0}}, "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1615104939, "i": 2}}, "signature": {"hash": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "subType": "00"}}, "keyId": 0}}}; nested exception is com.mongodb.MongoNotPrimaryException: Command failed with error 10107 (NotMaster): 'not master' on server localhost:60876. The full response is {"operationTime": {"$timestamp": {"t": 1615104939, "i": 1}}, "ok": 0.0, "errmsg": "not master", "code": 10107, "codeName": "NotMaster", "$gleStats": {"lastOpTime": {"$timestamp": {"t": 1615104939, "i": 1}}, "electionId": {"$oid": "7fffffff0000000000000001"}}, "lastCommittedOpTime": {"$timestamp": {"t": 0, "i": 0}}, "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1615104939, "i": 2}}, "signature": {"hash": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "subType": "00"}}, "keyId": 0}}}
    at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:133)
    at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2874)
    at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:568)
    at org.springframework.data.mongodb.core.MongoTemplate.saveDocument(MongoTemplate.java:1485)
    at org.springframework.data.mongodb.core.MongoTemplate.doSave(MongoTemplate.java:1421)
    at org.springframework.data.mongodb.core.MongoTemplate.save(MongoTemplate.java:1363)
....................
Caused by: com.mongodb.MongoNotPrimaryException: Command failed with error 10107 (NotMaster): 'not master' on server localhost:60876. The full response is {"operationTime": {"$timestamp": {"t": 1615104939, "i": 1}}, "ok": 0.0, "errmsg": "not master", "code": 10107, "codeName": "NotMaster", "$gleStats": {"lastOpTime": {"$timestamp": {"t": 1615104939, "i": 1}}, "electionId": {"$oid": "7fffffff0000000000000001"}}, "lastCommittedOpTime": {"$timestamp": {"t": 0, "i": 0}}, "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1615104939, "i": 2}}, "signature": {"hash": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "subType": "00"}}, "keyId": 0}}}
    at com.mongodb.internal.connection.ProtocolHelper.createSpecialException(ProtocolHelper.java:244)
    at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:171)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:302)

My test class has the following method.

@Test
   @Order(1)
   void testUpdateValidationsById() {
       MyEntity entity = new MyEntity();
       entity.setId("6030a6e1b32d1f70b4ce8039");
       entity.setAttributeType("IP Address, FQDN");
       entity.setCategory("NTP");
       entity.setCategoryLabel("NTP");
       entity.setTrueMsgDetails("It is true");
       entity.setFalseMsgDetails("It is false");
       entity.setCorevalidation("Core Validation");

       ResponseEntity<?> re = controller.updateValidationsById(entity);
       boolean flag = re.getStatusCode().is2xxSuccessful();
       assertEquals(true, flag);
   }

My test configuration class looks like this.

@Profile("test")
@ActiveProfiles("test")
@TestConfiguration
public class TestMongoDBConfig implements InitializingBean, DisposableBean {

    private MongodExecutable executable;
    private String replicaSetName = "rs0";
    private int mongoDBPort = 27021;

    @Override
    public void afterPropertiesSet() throws Exception {
        mongoDBPort = Network.getFreeServerPort();
        IMongoCmdOptions cmdOptions = new MongoCmdOptionsBuilder().useNoPrealloc(false).useSmallFiles(false)
                .master(false).verbose(false).useNoJournal(false).syncDelay(0).build();

//      IMongodConfig mongodConfig = new MongodConfigBuilder().version(Version.Main.PRODUCTION)
        IMongodConfig mongodConfig = new MongodConfigBuilder().version(Version.Main.V4_0)
//              .net(new Net(27023, Network.localhostIsIPv6()))
                .net(new Net(mongoDBPort, Network.localhostIsIPv6()))
                .replication(new Storage(null, replicaSetName, 5000)).configServer(true).cmdOptions(cmdOptions)
                .build();

        MongodStarter starter = MongodStarter.getDefaultInstance();
        executable = starter.prepare(mongodConfig);
        executable.start();
    }

    
    @Bean(name = "test1")
    public MongoClient mongoClient() {
        ConnectionString cs = new ConnectionString("mongodb://localhost:" + mongoDBPort + "/");
        MongoClient mongoClient = MongoClients.create(cs);
        System.out.println("--------------------------------------");
        System.out.println("Mongo Selected ort : " + mongoDBPort);
        System.out.println("MongoClient : " + mongoClient);
        System.out.println("--------------------------------------");
        mongoClient.getDatabase("admin").runCommand(new Document("replSetInitiate", new Document()));
        return mongoClient;
    }

    /**
     * Destroy.
     *
     * @throws Exception the exception
     */
    @Override
    public void destroy() throws Exception {
        executable.stop();
    }
}

Answer 1

As there is no solution for this I will post my solution, keep in mind this happens for an embedded mongo db(flapdoodle version 3.0.0) with replica set only and with the following java & spring mongo drivers:

        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-mongodb</artifactId>
            <version>2.2.13.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>mongo-java-driver</artifactId>
            <version>3.11.3</version>
        </dependency>

the solution we used is the following busy wait:

    private void waitUntilReplicaReady() throws InterruptedException {
        Document document = getRaplicaStatus();
        while (isReplicaInitializing(document)) {
            document = getRaplicaStatus();
            TimeUnit.MILLISECONDS.sleep(100);
        }
    }

    private boolean isReplicaInitializing(Document document) {
        return document.get("lastStableCheckpointTimestamp", BsonTimestamp.class).getTime() == 0;
    }

    private Document getRaplicaStatus() {
        return getDatabase("admin").runCommand(new BasicDBObject("replSetGetStatus", "1"));
    }

While lastStableCheckpointTimestamp time is not set we keep pooling, when lastStableCheckpointTimestamp is set we allow the program to advance(it takes ~2 seconds for it to populate)

Answer 2

Adding the read preference in connection string resolved above issue

mongodb://userName:Password@mongodb0.example.com:27017?ReadPreference=primary

Answer 3

Before change this, checking may help if IPs and ports are valid for the Mongo Db.