Using Spring @Scheduled and @Async together

Question

Here is my use case.

A legacy system updates a database queue table QUEUE.

I want a scheduled recurring job that - checks the contents of QUEUE - if there are rows in the table it locks the row and does some work - deletes the row in QUEUE

If the previous job is still running, then a new thread will be created to do the work. I want to configure the maximum number of concurrent threads.

I am using Spring 3 and my current solution is to do the following (using a fixedRate of 1 millisecond to get the threads to run basically continuously)

@Scheduled(fixedRate = 1)
@Async
public void doSchedule() throws InterruptedException {
    log.debug("Start schedule");
    publishWorker.start();
    log.debug("End schedule");
}

<task:executor id="workerExecutor" pool-size="4" />

This created 4 threads straight off and the threads correctly shared the workload from the queue. However I seem to be getting a memory leak when the threads take a long time to complete.

java.util.concurrent.ThreadPoolExecutor @ 0xe097b8f0                              |              80 |   373,410,496 |     89.74%
|- java.util.concurrent.LinkedBlockingQueue @ 0xe097b940                          |           48 |   373,410,136 |     89.74%
|  |- java.util.concurrent.LinkedBlockingQueue$Node @ 0xe25c9d68

So

1: Should I be using @Async and @Scheduled together?

2: If not then how else can I use spring to achieve my requirements?

3: How can I create the new threads only when the other threads are busy?

Thanks all!

EDIT: I think the queue of jobs was getting infinitely long... Now using

    <task:executor id="workerExecutor"
    pool-size="1-4"
    queue-capacity="10" rejection-policy="DISCARD" />

Will report back with results

Does it not work properly without the `@Async`? A method annotated with `@Scheduled` should be executed asynchronously anyway. — ach, Jan 31 '13 at 15:43
If you want "the threads to run continuously" then you should not really be using @Scheduled in the first place. Its use would be for "scheduled" activities, not continuous ones... — JoeG, May 07 '13 at 17:27
you might consider making publishWorker.start(); method Async. — fatih tekin, Jul 10 '13 at 13:49

score 4 · Answer 1 · answered Aug 27 '14 at 08:27

You can try

Run a scheduler with one second delay, which will lock & fetch all QUEUE records that weren't locked so far.
For each record, call an Async method, which will process that record & delete it.
The executor's rejection policy should be ABORT, so that the scheduler can unlock the QUEUEs that aren't given out for processing yet. That way the scheduler can try processing those QUEUEs again in the next run.

Of course, you'll have to handle the scenario, where the scheduler has locked a QUEUE, but the handler didn't finish processing it for whatever reason.

Pseudo code:

public class QueueScheduler {
    @AutoWired
    private QueueHandler queueHandler;

    @Scheduled(fixedDelay = 1000)
    public void doSchedule() throws InterruptedException {
        log.debug("Start schedule");
        List<Long> queueIds = lockAndFetchAllUnlockedQueues();
        for (long id : queueIds)
            queueHandler.process(id);
        log.debug("End schedule");
    }
}

public class QueueHandler {

    @Async
    public void process(long queueId) {
        // process the QUEUE & delete it from DB
    }
}
<task:executor id="workerExecutor" pool-size="1-4" queue-capcity="10"
     rejection-policy="ABORT"/>

If in any way you can provide pseudo code for ```lockAndFetchAllUnlockedQueues();```, it would be really helpful. — KnockingHeads, Oct 22 '21 at 11:19

score 1 · Answer 2 · edited May 23 '17 at 11:47

//using a fixedRate of 1 millisecond to get the threads to run basically continuously
@Scheduled(fixedRate = 1)

When you use @Scheduled a new thread will be created and will invoke method doSchedule at the specified fixedRate at 1 milliseconds. When you run your app you can already see 4 threads competing for the QUEUE table and possibly a dead lock.

Investigate if there is a deadlock by taking thread dump. http://helpx.adobe.com/cq/kb/TakeThreadDump.html

@Async annotation will not be of any use here.

Better way to implement this is to create you class as a thread by implementing runnable and passing your class to TaskExecutor with required number of threads.

Using Spring threading and TaskExecutor, how do I know when a thread is finished?

Also check your design it doesn't seem to be handling the synchronization properly. If a previous job is running and holding a lock on the row, the next job you create will still see that row and will wait for acquiring lock on that particular row.

Using Spring @Scheduled and @Async together

2 Answers2

Linked