I have a spring batch application which reads lines from a flat file, converts each line to an object, processes them, and writes them to a database. The steps are multi-threaded using spring's ThreadPoolTaskExecutor, but the performance is still terrible. Logging shows that reading is the main bottleneck, with a speed of 25 lines per second. Reading is multi-threaded and runs on 16 threads. But CPU utilization only shows 6 - 7% usage. The application is running on a 16-core 64GB RAM system, with a Tomcat Server. I suspect that it's not using all CPU cores. How can I configure Spring Batch to better utilize the CPU?
Step Configuration :
public Step createMyStep() {
try {
System.out.println("Creating MyStep");
AbstractTaskletStepBuilder builder = stepFactory.get("MyStep")
.<T, T>chunk(50)
.reader(multiResourceItemReader(null, null, null, null, null))
.listener(new ItemReadListenerImpl<>())
.processor(createProcessor())
.listener(new ItemProcessListenerImpl<>())
.writer(slaveStepWriter())
.faultTolerant()
.skip(RollbackException.class)
.skipLimit(1000000000)
.taskExecutor(taskExecutor())
.throttleLimit(64);
return builder.build();
}catch(Exception e){
throw e;
}
}
@Bean
public TaskExecutor taskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(16);
executor.setQueueCapacity(1);
executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
return executor;
}DataLoadConstants
The reader takes an InputStream as a resource, and uses FlatFileItemReader, configured with a lineMapper and BufferedReaderFactory to convert lines to an Object. The reader returns a SynchronizedItemStreamReader.
