JPA synchronization between micro-service instances

Question

I have a timer based job

@Component
public class Worker {

    @Scheduled(fixedDelay = 100)
    public void processEnvironmentActions() {
        Job job = pickJob();
    }

    public Job pickJob() {
        Job job = jobRepository.findFirstByStatus(Status.NOT_PROCESSED);
        job.setStatus(Status.PROCESSING);
        jobRepository.save(job);
        return job;
    }
}

Now, in the most situations this should give me correct result. But what will happen if there are two instances of microservice executing this piece of code at the same time?

How do I make sure that even if there are multiple instances of service, the repository should always give one job to only one instance and not other instances.

EDIT: I think people are getting confused/concentrated over @Transactional so removed it. The question remains the same.

Do you know that `@Transactional` doesn't work in case method called on `this`? — talex, Dec 18 '18 at 09:21
It should work if the method is `public`, right? In example, I've written it as a `private` method but if I make it `public` how do I achieve the synchronization between multiple instances? — Ganesh Satpute, Dec 18 '18 at 09:25
I don't know how to answer your question. I just notice problem in your code. — talex, Dec 18 '18 at 09:26
I think your question rather is: How do I implement a distributed worker queue across multiple service instances? — mp911de, Dec 18 '18 at 09:37
@talex is right. Transactional does not work with local calls because there is no proxy involved. To solve your problem you need to look the entire job table. I will add the answer later. — Simon Martinelli, Dec 18 '18 at 09:42
Have you taken a look at Shedlock? The method processEnvironmentActions() can be annotated with SchedulerLock annotation. https://www.baeldung.com/shedlock-spring — Hemant H Kumar, Feb 22 '22 at 10:41

score 5 · Answer 1 · answered Dec 18 '18 at 13:29

But what will happen if there are two instances of microservice executing this piece of code at the same time?

As so often the answer is: It depends.

All this assumes your code runs inside a transaction

Optimistic locking.

If your Job entity hast a version attribute, i.e. an attribute annotated with the @Version annotation. Optimistic locking is enabled. If to processes access the same job one will notice the versions attribute changed when trying to persist the changed job entity and fail with an OptimisticLockingException. All you have to do is handle that exception so you process doesn't die but tries again to get the next Job.
No (JPA level) locking.

If the Job entity doesn't have a version attribute, JPA will be default not apply any locking. The second process accessing a Job would issue an update, that essentially is a NOOP, since the first process already updated it. Neither will notice the problem. You probably want to avoid this.
Pessimistic locking

A pessimistic_write lock will prevent anyone from reading the entity before your done reading and writing it (at least that is my understanding of the JPA spec). Therefore this should avoid the second process to be able to select the row before the first process is done writing it. This probably blocks the whole second process. So make sure the transaction holding such a lock is short.

In order to obtain such a lock annotate the repository method findFirstByStatus with @Lock(LockModeType.PESSIMISTIC_WRITE).

Of course, there might be libraries or frameworks out there that handle these kinds of details for you.

Ganesh Satpute · Accepted Answer · 2018-12-24T09:33:30.323

@Jens Schauder 's answer points me in the right direction. Let me share the code so it guides other people. This is how I solved my problem, I changed my job class as below

@Entity 
public class Job {
   @Version
   private Long version = null; 
   // other fields omitted for bervity
}

Now, let's trace the following code

@Transactional
public Job pickJob() {
    Job job = jobRepository.findFirstByStatus(Status.NOT_PROCESSED);
    job.setStatus(Status.PROCESSING);
    Job saved jobRepository.save(job);
    return saved;
}

Note: Make sure you return the saved object and not the job object. If you return the job object, it'll fail for second save operation as the version count that was for job will be behind than that for saved.

.

Service 1                                 Service 2 

1. Read Object (version = 1)              1. Read Object (version = 1)
2. Change the object and save 
      (changes the version)
3. Continues to process                   2. Change the object and save 
                                               (this operation fails as 
                                                the version that was read 
                                                was 1 but in the DB version is 2)
                                          3. Skip the job processing

This way the job will picked up by only one process.

score 1 · Answer 3 · answered Dec 18 '18 at 09:41

1

I'm not familiar to spring-batch but apparently spring-batch implements optimistic locking so the save operation will fail if another thread already picked the same job.

See spring batch horizontal scaling

answered Dec 18 '18 at 09:41

Conffusion

4,335
2
16
28

score 1 · Answer 4 · answered Dec 18 '18 at 13:08

I agree with @Conffusion answer, but without knowing about the flush policy you should use jobRepository.saveAndFlush(job) method so that you are sure sql statements are pushed down to the database.

see also Difference between save and saveAndFlush in Spring data jpa

JPA synchronization between micro-service instances

4 Answers4