Spring Boot not able to update sharded collection on azure cosmos db(MongoDb)

Question

I have a collection "documentDev" present in the database with sharding key as 'dNumber' Sample Document :

{
"_id" : "12831221wadaee23",
"dNumber" : "115",
"processed": false
}

If I try to update this document through any query tool using a command like -

db.documentDev.update({
  "_id" : ObjectId("12831221wadaee23"),
  "dNumber":"115"
},{
    $set:{"processed": true}}, 
{ multi: false, upsert: false}
)}`

It updates the document properly. But if I do use spring boot's mongorepository command like DocumentRepo.save(Object) it throws an exception

Caused by: com.mongodb.MongoCommandException: Command failed with error 61: 'query in command must target a single shard key' on server by3prdddc01-docdb-3.documents.azure.com:10255. The full response is { "_t" : "OKMongoResponse", "ok" : 0, "code" : 61, "errmsg" : "query in command must target a single shard key", "$err" : "query in command must target a single shard key" }

This is my DocumentObject:

@Document(collection = "documentDev")
public class DocumentDev
{
@Id
private String id;
private String dNumber;
private String fileName;
private boolean processed;
}

This is my Repository Class -

@Repository
public interface DocumentRepo extends MongoRepository<DocumentDev, 
String> { }

and value i am trying to update

Value : doc : { "_id" : "12831221wadaee23", "dNumber" : "115", "processed": true }

the function I am trying to execute :

@Autowired
DocumentRepo docRepo;

docRepo.save(doc); // Fails to execute

Note: I have sharding enabled on dNumber field. And I am successfully able to update using Native queries on NoSQL Tool. I was also able to execute the Repository save operation on Non sharded collection.

Update: I am able to update the document by creating native query using MongoTemplate - My Query looks like this -

public DocumentDev updateProcessedFlag(DocumentDev request) {
    Query query = new Query();
    query.addCriteria(Criteria.where("_id").is(request.getId()));
    query.addCriteria(Criteria.where("dNumber").is(request.getDNumber()));
    Update update = new Update();
    update.set("processed", request.isProcessed());
    mongoTemplate.updateFirst(query, update, request.getClass());
    return request;
}

But this is not a generic solution as any other field might have update and my document may have other fields as well.

did you find a better solution ? I'm having the same issue but with the C# driver. — FrenchTastic, Mar 24 '19 at 16:08
There's a specific attribute in spring-data-mongodb:3.x that addresses this now. https://stackoverflow.com/questions/47538573/specifing-a-sharded-collection-with-spring-data-mongodb — PabloC, Oct 19 '20 at 08:10

score 1 · Answer 1 · edited Oct 29 '19 at 01:06

I had the same issue, solved with following hack:

@Configuration
public class ReactiveMongoConfig {

    @Bean
    public ReactiveMongoTemplate reactiveMongoTemplate(ReactiveMongoDatabaseFactory reactiveMongoDatabaseFactory,
            MongoConverter converter, MyService service) {
        return new ReactiveMongoTemplate(reactiveMongoDatabaseFactory, converter) {
            @Override
            protected Mono<UpdateResult> doUpdate(String collectionName, Query query, UpdateDefinition update,
                    Class<?> entityClass, boolean upsert, boolean multi) {
                query.addCriteria(new Criteria("shardKey").is(service.getShardKey()));
                return super.doUpdate(collectionName, query, update, entityClass, upsert, multi);
            }
        };
    }
}

Would be nice to have an annotation @ShardKey to mark document field as shard and have it added to query automatically.

score 0 · Answer 2 · edited Oct 28 '19 at 22:26

i am hacking this by create a custom repository:

public interface CosmosCustomRepository<T> {
    void customSave(T entity);
    void customSave(T entity, String collectionName);
}

the implement for this repository:

public class CosmosCustomRepositoryImpl<T> implements CosmosCustomRepository<T> {

    @Autowired
    private MongoTemplate mongoTemplate;

    @Override
    public void customSave(T entity) {
        mongoTemplate.upsert(createQuery(entity), createUpdate(entity), entity.getClass());
    }

    @Override
    public void customSave(T entity, String collectionName) {
        mongoTemplate.upsert(createQuery(entity), createUpdate(entity), collectionName);
    }

    private Update createUpdate(T entity) {
        Update update = new Update();
        for (Field field : entity.getClass().getDeclaredFields()) {
            try {
                field.setAccessible(true);
                if (field.get(entity) != null) {
                    update.set(field.getName(), field.get(entity));
                }
            } catch (IllegalArgumentException | IllegalAccessException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
        return update;
    }

    private Query createQuery(T entity) {
        Criteria criteria = new Criteria();
        for (Field field : entity.getClass().getDeclaredFields()) {
            try {
                field.setAccessible(true);
                if (field.get(entity) != null) {
                    if (field.getName().equals("id")) {
                        return new Query(Criteria.where("id").is(field.get(entity)));
                    }
                    criteria.and(field.getName()).is(field.get(entity));
                }
            } catch (IllegalArgumentException | IllegalAccessException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
        return new Query(criteria);
    }
}

your DocumentRepo will extends this new custom repository.

@Repository
public interface DocumentRepo extends MongoRepository<DocumentDev, String>, CosmosCustomRepository<DocumentDev> { }

To save new document, just use new customSave

@Autowired
DocumentRepo docRepo;

docRepo.customSave(doc);

score 0 · Answer 3 · answered Oct 28 '19 at 20:30

Following the custom repository approach, I got an error because spring is expecting a Cosmos entity to be available in the custom implementation {EntityName}CustomRepositoryImpl, so I renamed the implementation. I also added code for:

The case when entity has inherited fields
Shard key is not always the Id, we should add it along with the id: { "shardkeyName": "shardValue" }

Adding generated ObjectId to the entity for new documents

 public class DocumentRepositoryImpl<T> implements CosmosRepositoryCustom<T> {

    @Autowired
    protected MongoTemplate mongoTemplate;

    @Override
    public T customSave(T entity) {
        WriteResult writeResult = mongoTemplate.upsert(createQuery(entity), createUpdate(entity), entity.getClass());
        setIdForEntity(entity,writeResult);
        return entity;
    }

    @Override
    public T customSave(T entity, String collectionName) {
        WriteResult writeResult = mongoTemplate.upsert(createQuery(entity), createUpdate(entity), collectionName);
        setIdForEntity(entity,writeResult);
        return entity;
    }

    @Override
    public void customSave(List<T> entities) {
        if(CollectionUtils.isNotEmpty(entities)){
            entities.forEach(entity -> customSave(entity));
        }
    }

    public <T> Update createUpdate(T entity){
        Update update = new Update();
        for (Field field : getAllFields(entity)) {
            try {
                field.setAccessible(true);
                if (field.get(entity) != null) {
                    update.set(field.getName(), field.get(entity));
                }
            } catch (IllegalArgumentException | IllegalAccessException e) {
                LOGGER.error("Error creating update for entity",e);
            }
        }
        return update;
    }

    public <T> Query createQuery(T entity) {
        Criteria criteria = new Criteria();
        for (Field field : getAllFields(entity)) {
            try {
                field.setAccessible(true);
                if (field.get(entity) != null) {
                    if (field.getName().equals("id")) {
                        Query query = new Query(Criteria.where("id").is(field.get(entity)));
                        query.addCriteria(new Criteria(SHARD_KEY_NAME).is(SHARD_KEY_VALUE));
                        return query;
                    }
                    criteria.and(field.getName()).is(field.get(entity));
                }
            } catch (IllegalArgumentException | IllegalAccessException e) {
                LOGGER.error("Error creating query for entity",e);
            }
        }
        return new Query(criteria);
    }

    private <T> List<Field> getAllFields(T entity) {
        List<Field> fields = new ArrayList<>();
        fields.addAll(Arrays.asList(entity.getClass().getDeclaredFields()));
        Class<?> c = entity.getClass().getSuperclass();
        if(!c.equals(Object.class)){
            fields.addAll(Arrays.asList(c.getDeclaredFields()));
        }
        return fields;
    }

    public <T> void setIdForEntity(T entity,WriteResult writeResult){
        if(null != writeResult && null != writeResult.getUpsertedId()){
            Object upsertId = writeResult.getUpsertedId();
            entity.setId(upsertId.toString());
        }
    }
}

I am using spring-boot-starter-mongodb:1.5.1 with spring-data-mongodb:1.9.11

score 0 · Answer 4 · answered Sep 30 '22 at 11:26

A more recent and simple approach I've figured out recently is to use the @Sharded({shardKey1, shardKey2, ...}) spring annotation like so:

@Document(collection = "documentDev")
@Sharded(shardKey = {"dNumber"})
public class DocumentDev {
  @Id private String id;
  private String dNumber;
  private String fileName;
  private boolean processed;
}

This is going to be picked up by MongoRepository automatically. And here is the doc: https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#sharding

Enjoy coding!

Spring Boot not able to update sharded collection on azure cosmos db(MongoDb)

4 Answers4

Linked