0

We are changing our spring batch application to multiple data source so that metadata tables can go to H2 database(Avoid operational purging effort and improve performance) and business tables will be in oracle DB. After changing this, CompositeWriter transaction is partially committing as we have 3 tables in CompositeWriter. Ist table committed and 2nd table error thrown but rollback not happening for Ist table. This is working fine in default single data source . This question is related to transaction management of multiple datasources in spring batch having composite writer.

    ######### SPRING BATCH H2 DATASOUCRE Batch MetData ######
    springbatch.datasource.driver-class-name=org.h2.Driver
    springbatch.datasource.url=jdbc:h2:file:C:/IDE/jan
    springbatch.datasource.username=ngecom
    springbatch.datasource.password=ngecom
    springbatch.datasource.configuration.connection-timeout=600000
    springbatch.datasource.configuration.minimum-idle=5 
    springbatch.datasource.configuration.maximum-pool-size=50
    springbatch.datasource.configuration.idle-timeout=600000 
    springbatch.datasource.configuration.max-lifetime=1800000 
    springbatch.datasource.configuration.auto-commit=true 
    springbatch.datasource.configuration.poolName=SpringBoot-HikariCPPrimary
    springbatch.datasource.configuration.leak-detection-threshold=0
    
    ########JDBC Oracle Datasource########
    #connection timeout 10 min
    app.datasource.username=ngbilling
    app.datasource.url=ourDBServer
    app.datasource.driver-class-name=oracle.jdbc.OracleDriver
    app.datasource.configuration.connection-timeout=600000
    app.datasource.configuration.minimum-idle=5 
    app.datasource.configuration.maximum-pool-size=50
    app.datasource.configuration.idle-timeout=600000 
    app.datasource.configuration.max-lifetime=1800000 
    app.datasource.configuration.auto-commit=true 
    app.datasource.configuration.poolName=SpringBoot-HikariCPSecodary
    app.datasource.configuration.leak-detection-threshold=0
    
    
    @Configuration
    public class DataSourceRepository {
    
        @Bean   
        @Primary
        @ConfigurationProperties(prefix = "springbatch.datasource")
        public DataSourceProperties springBatchDataSourceProperties() {
            return new DataSourceProperties();
        }
    
        @Bean(name = {"dataSourceSpringBatch"})
        @Primary
        @ConfigurationProperties("springbatch.datasource.configuration")
        public HikariDataSource springBatchDataSource() {
            return springBatchDataSourceProperties().initializeDataSourceBuilder().type(HikariDataSource.class).build();
        }
        
        
        @Bean   
        @ConfigurationProperties(prefix = "app.datasource")
        public DataSourceProperties appDataSourceProperties() {
            return new DataSourceProperties();
        }
    
        @Bean(name = {"bscsDataSource"})
        @ConfigurationProperties("app.datasource.configuration")
        public HikariDataSource appDataSource() {
            return appDataSourceProperties().initializeDataSourceBuilder().type(HikariDataSource.class).build();
        }
        
        @Bean(name = "bscsJDBCTemplate")
        public JdbcTemplate jdbcTemplate(@Qualifier("bscsDataSource") DataSource bscsDataSource) {
         return new JdbcTemplate(bscsDataSource);
    }
    
      @Bean
        public CompositeItemWriter compositeWriter() throws Exception {
            CompositeItemWriter compositeItemWriter = new CompositeItemWriter();
            List<ItemWriter> writers = new ArrayList<ItemWriter>();
            writers.add(summaryWriter());
            writers.add(detailsWriter());
            writers.add(chartWriter());
            compositeItemWriter.setDelegates(writers);
            return compositeItemWriter;
        }   
    
    @Bean
        public JdbcBatchItemWriter<UnifiedInvoiceDTO> summaryWriter() {
            JdbcBatchItemWriter<UnifiedInvoiceDTO> databaseItemWriter = new JdbcBatchItemWriter<>();
            databaseItemWriter.setDataSource(dataSource);
            databaseItemWriter.setSql(BSCSUtils.QUERY_INSERT_UBI_SUMMARY);
            ItemPreparedStatementSetter<UnifiedInvoiceDTO> invoicePreparedStatementSetter = new InvoiceSummarySetter();
            databaseItemWriter.setItemPreparedStatementSetter(invoicePreparedStatementSetter);
            return databaseItemWriter;
        }
     @Bean
        public Step slaveStep() throws Exception {
            return stepBuilderFactory.get("slaveStep").<UnifiedInvoiceDTO, UnifiedInvoiceDTO>chunk(chunkSize)
                    .reader(xmlItemReader(null)).processor(xmlFileItemProcessor()).writer(compositeWriter()).faultTolerant()
                    .skipLimit(skipErrorCount).skip(Exception.class).noSkip(FileNotFoundException.class)
                    .listener(itemSkipListener()).build();
        }

@Component
public class BatchJobScheduler extends BasicBatchConfigurer {
    protected BatchJobScheduler(BatchProperties properties, DataSource dataSource,
            TransactionManagerCustomizers transactionManagerCustomizers) {
        super(properties, dataSource, transactionManagerCustomizers);
        // TODO Auto-generated constructor stub
    }}
Rakesh
  • 658
  • 6
  • 15
  • Does this answer your question? [How to java-configure separate datasources for spring batch data and business data? Should I even do it?](https://stackoverflow.com/questions/25256487/how-to-java-configure-separate-datasources-for-spring-batch-data-and-business-da) – Mahmoud Ben Hassine Jan 12 '21 at 11:33
  • @Mahmoud Multiple datasource is working fine for me but if I add multiple data source "COMPOSITEWRITER" delegates part of business datasource doing partial commit. It's related to transactions that i am missing some where in configuration. – Rakesh Jan 12 '21 at 15:19
  • If your transaction spans multiple datasources, you need to configure a transaction manager for each datasource and use them as delegates of a `JtaTransactionManager`. The `JtaTransactionManager` should then be configured in your step. – Mahmoud Ben Hassine Jan 13 '21 at 08:19
  • Transaction is not spanning across multiple data source. Its on same data source. Do we still require specific transaction manager for each data source other than default. First Data source is for H2 to maintain batch tables. Second DataSource is for Oracle having a CompositeWriter have multiple delegates, Here is the issue second writer failed Ist one is not roll backing. – Rakesh Jan 13 '21 at 08:24
  • 1
    ok understood. In that case, you need to configure the step with a transaction manager for oracle. The job repository could be configured with a transaction manager for H2. Now if your composite writer delegates to 3 writers that write to 3 tables of the *same* datasource, then there will be a single transaction around the composite (it's all or nothing for the 3 delegates) and if one of the writers fails, the transaction will be rolled back for all. I shared a complete example here: https://stackoverflow.com/a/51918357/5019386. – Mahmoud Ben Hassine Jan 13 '21 at 08:51
  • @Mahmoud Our current application is spring batch single default data source and this issue is not there and rollback happening in COMPOSITEWRITER perfectly. When we add multiple data source only this issue is happening. https://stackoverflow.com/a/51918357/5019386 doesnt have separate transaction managers. Let us create separate transaction managers for each datasource and will update you. – Rakesh Jan 13 '21 at 18:00
  • @Mahmoud :- Thanks a lot it works with separate transaction manager for secondary datasource that used for Oracle chunk based Composite writer. – Rakesh Mar 19 '21 at 17:01

0 Answers0