Unable to limit the parallelism using ThreadPoolExecutor with Spring Batch

Question

Here's my configuration:


    @StepScope
    @Bean(name = "mySlaveStep")
    public Step mySlaveStep(
            @Qualifier(value = "myReader") ItemReader reader,
            @Qualifier(value = "myWriter") ItemWriter writer,
            StepBuilderFactory stepBuilderFactory) {
        return stepBuilderFactory.get("MySlaveStep")
                .<SomeObject, SomeObject>chunk(1000)
                .reader(reader)
                .writer(writer)
                .build();
    }

    @Bean(name = "myStep")
    public Step myStep(
            @Qualifier(value = "myPartitioner") Partitioner partitioner, // with @StepScope
            @Qualifier(value = "myExecutor") TaskExecutor executor, // With/without @StepScope
            @Qualifier(value = "myStep") Step step, // With @StepScope
            StepBuilderFactory stepBuilderFactory) {
        return stepBuilderFactory
                .get("MyStep")
                .partitioner("MyPartition", partitioner)
                .taskExecutor(executor)
                .step(step)
                .build();
    }

    @StepScope // With or without
    @Bean(name = "taskExecutor")
    public TaskExecutor taskExecutor() {
        final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setQueueCapacity(Integer.MAX_VALUE);
        executor.setCorePoolSize(2);
        executor.setMaxPoolSize(2);
        return executor;
    }

The exception I'm getting is:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.mySlaveStep': Scope 'step' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No context holder available for step scope
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:368) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35) ~[spring-aop-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:192) ~[spring-aop-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at com.sun.proxy.$Proxy144.execute(Unknown Source) ~[na:na]
    at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:138) ~[spring-batch-core-4.2.0.RELEASE.jar:4.2.0.RELEASE]
    at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:135) ~[spring-batch-core-4.2.0.RELEASE.jar:4.2.0.RELEASE]
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: java.lang.IllegalStateException: No context holder available for step scope
    at org.springframework.batch.core.scope.StepScope.getContext(StepScope.java:167) ~[spring-batch-core-4.2.0.RELEASE.jar:4.2.0.RELEASE]
    at org.springframework.batch.core.scope.StepScope.get(StepScope.java:99) ~[spring-batch-core-4.2.0.RELEASE.jar:4.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:356) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]

So I assume this is related to this. Removing/adding @StepScope from TaskExecutor bean does not change the outcome, however, removing TaskExecutor altogether resolves the issue. I'm only trying to limit the number of parallel partitions being handled as per here. How do I go about it?

I added an answer about bean scopes. But in order to correctly answer the question about limiting the parallelism using a ThreadPoolTaskExecutor, I need to know which implementation of `PartitionHandler` you are using. Is it the `TaskExecutorPartitionHandler`? This is key to answer that question, but is not mentioned in the code you shared. — Mahmoud Ben Hassine, Jan 27 '23 at 08:28
I'm not using a specific `PartitionHandler`, not configuring it at all. The whole configuration is as in the question. So do I use `TaskExecutorPartitionHandler` and pass my `taskExecutor()` to it? — Hasan Can Saral, Jan 27 '23 at 08:32
ok, so by default you will get a `TaskExecutorPartitionHandler` from the builder, configured with your task executor. With that in place, any limit on the number of threads you set on the task executor should be applied/respected by the partitioned step as expected. I see you accepted the answer. Glad it helped! — Mahmoud Ben Hassine, Jan 27 '23 at 09:26
@MahmoudBenHassine I understand, but the real issue was related to autowiring, which is your actual answer, so thanks for that. — Hasan Can Saral, Jan 30 '23 at 09:41

score 1 · Accepted Answer · answered Jan 27 '23 at 08:25

First, I will address the scope of the task executor. The task executor should not be "SpringBatch-scoped" (job-scoped or step-scoped) or even scoped at all (IMO the default singleton scope is the correct scope of such a compnent for most use cases). Spring Batch does not create or manage threads, it delegates that to task executors in different parts of the framework. Therefore, such a component should not be impacted by any scope of Spring Batch and should not impact the behaviour of a Spring Batch job by any mean. If this is the case, that would be a bug in Spring Batch.

Now let me address the scope of a step. A step in Spring Batch cannot be step-scoped. That does not make sense. Marking the step as step-scoped means do not create that step bean until the job enclosing it is running (ie at runtime). But, at that time, the step was not configured yet. A batch artefact of a step (reader, writer, listener, tasklet, partitioner, etc) can be step-scoped though, but not the step itself. There is a note about that in the reference documentation here: Late Binding of Job and Step Attributes. Removing the step scope on mySlaveStep should fix your issue.

While I see valid use cases for step components to be scoped (to use late-binding for instance), I do not see any valid use case to scope the step itself.

Removing `@StepScope` from `mySlaveStep`, making `TaskExecutor` not step scoped and using `TaskExecutorPartitionHandler` resolved the issue. — Hasan Can Saral, Jan 27 '23 at 09:06

Unable to limit the parallelism using ThreadPoolExecutor with Spring Batch

1 Answers1