Spring-Batch without persisting metadata to database?

Question

I want to create a spring-batch job, but I want to run it without any database persistence. Unfortunately spring-batch requires to write metadata ob the job cycles to a database somehow, thus procing me to provide at least some kind of db with transactionmanager and entitymanager.

It it possible to prevent the metadata and run independently from txmanagers and databases?

Update:

ERROR org.springframework.batch.core.job.AbstractJob: Encountered fatal error executing job
java.lang.NullPointerException
    at org.springframework.batch.core.repository.dao.MapJobExecutionDao.synchronizeStatus(MapJobExecutionDao.java:158) ~[spring-batch-core-3.0.1.RELEASE.jar:3.0.1.RELEASE]
    at org.springframework.batch.core.repository.support.SimpleJobRepository.update(SimpleJobRepository.java:161) ~[spring-batch-core-3.0.1.RELEASE.jar:3.0.1.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_51]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98) ~[spring-tx-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262) ~[spring-tx-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95) ~[spring-tx-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at com.sun.proxy.$Proxy134.update(Unknown Source) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_51]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) ~[spring-batch-core-3.0.1.RELEASE.jar:3.0.1.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at com.sun.proxy.$Proxy134.update(Unknown Source) ~[?:?]
    at org.springframework.batch.core.job.AbstractJob.updateStatus(AbstractJob.java:416) ~[spring-batch-core-3.0.1.RELEASE.jar:3.0.1.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:299) [spring-batch-core-3.0.1.RELEASE.jar:3.0.1.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) [spring-batch-core-3.0.1.RELEASE.jar:3.0.1.RELEASE]
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) [spring-batch-core-3.0.1.RELEASE.jar:3.0.1.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_51]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) [spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-3.0.1.RELEASE.jar:3.0.1.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) [spring-aop-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at com.sun.proxy.$Proxy50.run(Unknown Source) [?:?]

Can you share your job and job repository configuration (assuming you're using the MapJobRepository that user3218114 mentions)? — Michael Minella, Aug 06 '14 at 15:38

score 64 · Answer 1 · answered Mar 08 '17 at 17:10

64

Simply create a configuration without datasource for Batch configuration :

@Configuration
@EnableAutoConfiguration
@EnableBatchProcessing
public class BatchConfiguration extends DefaultBatchConfigurer {

    @Override
    public void setDataSource(DataSource dataSource) {
        // override to do not set datasource even if a datasource exist.
        // initialize will use a Map based JobRepository (instead of database)
    }

}

It will initialize JobRepository and JobExplorer with a in memory map based implementation. https://github.com/spring-projects/spring-batch/blob/342d27bc1ed83312bdcd9c0cb30510f4c469e47d/spring-batch-core/src/main/java/org/springframework/batch/core/configuration/annotation/DefaultBatchConfigurer.java#L84

and you can use you're production datasource as well even if auto configured with spring boot.

answered Mar 08 '17 at 17:10

Aure77

3,034
7
33
53

I am using a datasource to extract data and transforms it with a data source, and Spring Batch was using this datasource to store his tables. Now I can use my Datasource and there are Spring Batch Tables but they are not used. Thanks. – AntuanSoft Dec 28 '17 at 00:29
1

You can combine with spring.batch.initializer.enabled=false in your applicaton.properties in order to not recreate the Spring Batch tables. – AntuanSoft Dec 28 '17 at 00:34
1

Nice solution thank you, thanks to @AntuanSoft also. – v.ladynev Feb 24 '18 at 23:50
The best Answer for me. – Faraz Nov 10 '18 at 00:59
But this config awaiting for dataSource. I want to work without dataSource at all – gstackoverflow Aug 19 '19 at 10:40
3

unfortunately this disabled persistence not only for spring batch but for the entire project – Sepultura Feb 10 '21 at 12:17

Braj · Answer 2 · 2016-09-10T18:22:09.727

31

I want to run it without any database persistence

You can use MapJobRepositoryFactoryBean and ResourcelessTransactionManager

sample configuration:

<bean id="transactionManager"
    class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

<bean id="jobRepository"
    class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
    <property name="transactionManager" ref="transactionManager" />
</bean>

<bean id="jobLauncher"
    class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository" />
</bean>

For Spring 4.X, the annotation based configuration would be as follows:

@Bean
public PlatformTransactionManager getTransactionManager() {
    return new ResourcelessTransactionManager();
}

@Bean
public JobRepository getJobRepo() {
    return new MapJobRepositoryFactoryBean(getTransactionManager()).getObject();
}

edited Sep 10 '16 at 18:22

answered Aug 01 '14 at 10:01

Braj

46,415
5
60
76

Unfortunately that produces the error updated above: NPE during `MapJobExecutionDao.synchronizeStatus()`... – membersound Aug 05 '14 at 13:03
1

should I share you complete configuration file? – Braj Aug 05 '14 at 13:06
1

Maybe that might help, that would be very kind. My configuration looks exactly like this: https://github.com/lhinze/spring-batch-retrytemplate-example/blob/master/src/main/java/example/ExampleConfiguration.java The main problem seems to be that in `MapJobExecutionDao.class` the `getJobExecution()` method returns null, as `executionsById` map is empty. Do I maybe might have forgotten to do any initializations? – membersound Aug 05 '14 at 13:21
I came back to this, and migrated your sample configuration to spring 4 annotation based config. Edited above, and it works as expected! – membersound Sep 04 '14 at 09:26
3

Tried this an ti is not working with spring-boot + spring-batch – Majky Oct 07 '16 at 09:40
@Braj You mean : use "MapJobRepositoryFactoryBean" and "ResourcelessTransactionManager" instead of "JobBuilderFactory" for example ? (Is jobbuilderfactory is responsible of doing persistence ?) – Sushi Jan 15 '18 at 17:07
The above configuration is posted on Aug 2014 in the context of core spring framework only. I am not sure whether it will work with spring boot or not. – Braj Jan 11 '19 at 03:46
Same here, it fails. The only one that worked without raising the NPE was extending `DefaultBatchConfigurer` and to override `setDataSource(DataSource dataSource) by calling `super.setDataSource(null);` (used with spring-batch-core 4.1.2, spring-boot 2.1.6, spring 5.1.8. – belgoros Jul 05 '19 at 15:23

score 18 · Answer 3 · answered Dec 03 '14 at 09:20

18

After tweaking @Braj's answer, my working configuration looks as follows:

@Bean
public ResourcelessTransactionManager transactionManager() {
    return new ResourcelessTransactionManager();
}

@Bean
public JobRepository jobRepository(ResourcelessTransactionManager transactionManager) throws Exception {
    MapJobRepositoryFactoryBean mapJobRepositoryFactoryBean = new MapJobRepositoryFactoryBean(transactionManager);
    mapJobRepositoryFactoryBean.setTransactionManager(transactionManager);
    return mapJobRepositoryFactoryBean.getObject();
}

@Bean
public SimpleJobLauncher jobLauncher(JobRepository jobRepository) {
    SimpleJobLauncher simpleJobLauncher = new SimpleJobLauncher();
    simpleJobLauncher.setJobRepository(jobRepository);
    return simpleJobLauncher;
}

answered Dec 03 '14 at 09:20

Cleankod

2,220
5
32
52

For me, somehow xml based configuration works but annotation based doesn't. – Sabir Khan Aug 17 '16 at 10:30
Are you sure that the class to wchich you copied these `@Bean` definitions is annotated with `@Configuration` and that it is being picked up by the Spring Boot? See the startup logs to find out. – Cleankod Dec 16 '16 at 10:51
It works provided I explicitly exclude datasource beans as answered by me in [this question](http://stackoverflow.com/questions/39359840/spring-boot-batch-resourcelesstransactionmanager-datasourcepropertiesdatasource/39366886#39366886) – Sabir Khan Jan 03 '17 at 04:30
Thanks @Sabir Khan @SpringBootApplication(exclude={DataSource.class,DataSourceAutoConfiguration.class}) works for me. – Kishor K Mar 26 '17 at 02:27
1

Cleankod's configuration causes `org.springframework.transaction.TransactionSuspensionNotSupportedException` to be thrown for my Batch processing using Spring Boot 1.5.2. – oschlueter Apr 24 '17 at 12:08
@Cleankod- how to set up job with JobParameters – Jeff Cook Aug 25 '18 at 17:25

score 14 · Accepted Answer · answered May 02 '17 at 15:42

14

I came back to my own question, as the solution did not work anymore. As of spring-batch-1.5.3 use as follows:

@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class})
...
    @Bean
    public PlatformTransactionManager transactionManager() {
        return new ResourcelessTransactionManager();
    }
}

answered May 02 '17 at 15:42

membersound

81,582
193
585
1,120

1

Thanks for this. I did not need to declare transactionManager and it just worked fine with exclude. – Rakesh Jan 04 '18 at 15:07
2

Is this in assumption with no other database being used for application? – Abhi Mar 06 '19 at 06:44
1

This is not working in case your batch using jpa repository to save data into db – DanzerZoneJS Sep 18 '20 at 16:50
Thanks Bro, as i am not using any DB this is useful – Shridutt Kothari Jun 26 '22 at 09:21

score 9 · Answer 5 · answered Jul 06 '17 at 07:49

If you don't want to store job's metadata in the database, configuration looks as follows

@Configuration
public class InMemoryJobRepositoryConfiguration {
@Bean
public ResourcelessTransactionManager transactionManager() {
    return new ResourcelessTransactionManager();
}

@Bean
public MapJobRepositoryFactoryBean mapJobRepositoryFactory(ResourcelessTransactionManager transactionManager)
        throws Exception {
    MapJobRepositoryFactoryBean factory = new MapJobRepositoryFactoryBean(transactionManager);
    factory.afterPropertiesSet();
    return factory;
}

@Bean
public JobRepository jobRepository(MapJobRepositoryFactoryBean repositoryFactory) throws Exception {
    return repositoryFactory.getObject();
}

@Bean
public JobExplorer jobExplorer(MapJobRepositoryFactoryBean repositoryFactory) {
    return new SimpleJobExplorer(repositoryFactory.getJobInstanceDao(), repositoryFactory.getJobExecutionDao(),
            repositoryFactory.getStepExecutionDao(), repositoryFactory.getExecutionContextDao());
}

@Bean
public SimpleJobLauncher jobLauncher(JobRepository jobRepository) {
    SimpleJobLauncher launcher = new SimpleJobLauncher();
    launcher.setJobRepository(jobRepository);
    return launcher;
}
}

If job needs to read/write business data to the database and the datasource is configured as follows.

@Autowired
private DataSource dataSource;

spring batch creates internal schema (BATCH_* tables and sequences) using that datasource. To prevent this from happening set flag spring.batch.initializer.enabled=false in application.properties.

`spring.batch.initializer.enabled=false` is now `spring.batch.jdbc.initialize-schema=never` — albertocavalcante, Mar 04 '22 at 08:28

score 7 · Answer 6 · edited Jun 15 '18 at 22:40

I tried all of the solutions above, but it does not work with my particular scenario because my spring batch job has advanced features such as multi-threading. It needs a database. So here is what I did to solve the problem:

@Configuration
public class FakeBatchConfig implements BatchConfigurer {

  PlatformTransactionManager transactionManager;
  JobRepository jobRepository;
  JobLauncher jobLauncher;
  JobExplorer jobExplorer;

  @Override
  public JobRepository getJobRepository() {
    return jobRepository;
  }

  @Override
  public PlatformTransactionManager getTransactionManager() {
    return transactionManager;
  }

  @Override
  public JobLauncher getJobLauncher() {
    return jobLauncher;
  }

  @Override
  public JobExplorer getJobExplorer() {
    return jobExplorer;
  }

  @PostConstruct
  void initialize() throws Exception {

    if (this.transactionManager == null) {
      this.transactionManager = new ResourcelessTransactionManager();
    }

    MapJobRepositoryFactoryBean jobRepositoryFactory = new MapJobRepositoryFactoryBean(this.transactionManager);
    jobRepositoryFactory.afterPropertiesSet();
    this.jobRepository = jobRepositoryFactory.getObject();

    MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(jobRepositoryFactory);
    jobExplorerFactory.afterPropertiesSet();
    this.jobExplorer = jobExplorerFactory.getObject();
    this.jobLauncher = createJobLauncher();
  }

  private JobLauncher createJobLauncher() throws Exception {
    SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
    jobLauncher.setJobRepository(jobRepository);
    jobLauncher.afterPropertiesSet();
    return jobLauncher;
  }

}

Thanks Jeff for this edit. My original post was written in Groovy, not Java. — Tracy Xia, Jun 22 '18 at 16:54
@Qing Xia, Thanks. Could you please guide how we can pass the JobParameters here ? — Jeff Cook, Aug 25 '18 at 17:14
When I use the same solution I get a problem using Spring Data Repository.save() .. it doesn't persist objects ! — Ghassen, Sep 23 '18 at 16:18
@Ghassen I am facing issue that save is not working. Any pointer how you solved it? — user2800089, Mar 24 '20 at 03:12
@Ghassen I am also facing same issue. Batch is not being able to save data to db. — DanzerZoneJS, Sep 18 '20 at 16:47

score 6 · Answer 7 · answered May 16 '17 at 09:04

6

In addition to @Braj-s response I had to exclude Batch autoconfiguration: @SpringBootApplication(exclude = {BatchAutoConfiguration.class})

It was not working otherwise. This includes Spring Boot application

answered May 16 '17 at 09:04

gmode

3,601
4
31
39

score 2 · Answer 8 · answered Jul 30 '18 at 14:25

After reading all the answers provided I got it working. I used mix of membersound and Aure77 solution. I found out there was no need to override setDataSource method. DefaultBatchConfigurer automatically takes care of PlatformTransactionManager and DataSource is not required. Please note I am using Spring boot 2.0.3.RELEASE. Following is the method I used

@SpringBootApplication(exclude = { DataSourceAutoConfiguration.class })
@EnableBatchProcessing
public class DemoApplication extends DefaultBatchConfigurer {

    @Autowired
    private JobBuilderFactory jobs;

    @Autowired
    private StepBuilderFactory steps;

    @Bean
    protected Tasklet tasklet() {

        return new Tasklet() {
            @Override
            public RepeatStatus execute(StepContribution contribution, ChunkContext context) {
                return RepeatStatus.FINISHED;
            }
        };
    }

    @Bean
    public Job job() throws Exception {
        return this.jobs.get("job").start(step1()).build();
    }

    @Bean
    protected Step step1() throws Exception {
        return this.steps.get("step1").tasklet(tasklet()).build();
    }

    public static void main(String[] args) throws Exception {
        //System.exit(SpringApplication.exit(SpringApplication.run(DemoApplication.class, args)));
        SpringApplication.run(DemoApplication.class, args);
    }
}

score 2 · Answer 9 · answered Aug 01 '18 at 15:21

2

The simplest of all is add an embedded database to classpath. Naturally in production is not recommended but if you're using it to learn, save time.

Add to your pom.xml the embedded H2 database dependency:

<dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
    <scope>runtime</scope>
</dependency>

That's all.

answered Aug 01 '18 at 15:21

bl4ckr0se

606
8
15

1

Yes, this is definitely the simplest! Thank you. – ben3000 Sep 14 '18 at 02:14
Note that the MapJobRepositoryFactoryBean will be removed in Spring Batch v5. As a result, the majority of answers in this post will no longer be correct. Spring Batch maintainers have expressed that only embedded databases are supported moving forward: https://docs.spring.io/spring-batch/docs/current/reference/html/job.html#inMemoryRepository – Michael Hoffman Sep 03 '22 at 21:18
Some supporting documentation to the change @MichaelHoffman was referring to: https://github.com/spring-projects/spring-batch/issues/3834 – Simeon Leyzerzon Aug 22 '23 at 17:48

score 0 · Answer 10 · answered Dec 20 '19 at 14:53

0

Following changes can be made to use the Spring Batch without having to create spring batch related metadata database structures.

This worked for me:

Added bean of type ResourcelessTransactionManager and configured MapJobRepositoryFactoryBean.
Using the configured bean MapJobRepositoryFactoryBean created a SimpleJoblauncher bean.
Added annotation @EnableBatchProcessing below @SpringBootApplication.
To override the defaults I had to add spring.main.allow-bean-definition-overriding=true in application.properties.

answered Dec 20 '19 at 14:53

rv.comm

675
1
7
10

I Just wanted to use spring batch provided features like Scheduler and ItemReader and ItemWriter, but did not wanted to maintain execution status in database tables as the requirement I had is a simple and insignificant one. If you have a complex enterprise level requirement using all the spring batch tables will be helpful and I would encourage doing so. – rv.comm Jun 02 '20 at 12:34

Spring-Batch without persisting metadata to database?

10 Answers10

Linked