-1

I have to write a batch job which will have as its source, a Teradata DB & as target, a Postgres DB using Spring batch framework. Could anyone let me know or refer me to a tutorial on how to achieve this, if at all its possible?

If not, what are the other options I could explore in order to do this?

UPDATE

I have a spring batch job, trying to make it work with Oracle(Source) & Postgres(Target), later I will be changing the source to Teradata. However, I am getting below error, which i assume is due to, the fact that job sequences are being looked up in the Oracle DB(where the sequences cant be created, as the DB user has Read Only access). How do I make the Job, create the sequences in the target DB(Postgres)?

Errors:

at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:227)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:121)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:115)
at org.springframework.boot.SpringApplication.runCommandLineRunners(SpringApplication.java:672)
... 5 common frames omitted
Caused by: java.sql.SQLException: ORA-02289: sequence does not exist

at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:745)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:210)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:804)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1049)
at oracle.jdbc.driver.T4CStatement.executeMaybeDescribe(T4CStatement.java:845)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1154)
at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1313)
at org.springframework.jdbc.support.incrementer.AbstractSequenceMaxValueIncrementer.getNextKey(AbstractSequenceMaxValueIncrementer.java:69)
... 40 common frames omitted

2016-08-12 12:34:57.467  INFO 11660 --- [main]    s.c.a.AnnotationConfigApplicationContext : Closing   org.springframework.context.annotation.AnnotationConfigApplicationContext@1cea91    b: startup date [Fri Aug 12 12:34:22 IST 2016]; root of context hierarchy
2016-08-12 12:34:57.469  INFO 11660 --- [main] o.s.j.e.a.AnnotationMBeanExporter          : Unregistering JMX-exposed beans on shutdown

** BatchConfiguration.java **

@Configuration
@EnableBatchProcessing
@ComponentScan(basePackageClasses = MyBatchConfigurer.class)
public class BatchConfiguration {

@Autowired
public JobBuilderFactory jobBuilderFactory;

@Autowired
public StepBuilderFactory stepBuilderFactory;

@Autowired
@Qualifier("oracleDBDataSource")
public DataSource dataSource1; 

@Autowired 
@Qualifier("postgresDataSource")
public DataSource dataSource2; 



//Read an Event from oracle DB
@Bean
public ItemReader<Event> reader(){

    JdbcCursorItemReader<Event> databaseReader = new JdbcCursorItemReader<>();
    String sqlQ = "select * from test_event";
    databaseReader.setDataSource(dataSource1);
    databaseReader.setSql(sqlQ);
    databaseReader.setRowMapper(new BeanPropertyRowMapper<>(Event.class));      
    return databaseReader;
}
//EventItemProcessor
@Bean
public EventItemProcessor processor() {

    return new EventItemProcessor();
}

@Bean
public JdbcBatchItemWriter<Event> writer() {

    JdbcBatchItemWriter<Event> writer = new JdbcBatchItemWriter<Event>();
    writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Event>());
    writer.setSql("insert into EVENTS (event_id, event_title, event_start_date, event_end_date, event_type,need_offline_sync, event_status, created_by, created_date, last_updated_by,last_updated_date) VALUES (:event_id, :event_title, :event_start_date, :event_end_date, :event_type,:need_offline_sync, :event_status, :created_by, :created_date, :last_updated_by, :last_updated_date)");
    writer.setDataSource(dataSource2);
    return writer;
}
//Postgres Data source for Target
@Bean(name="postgresDataSource")     
public DataSource postgresDataSource() throws SQLException {

    final SimpleDriverDataSource dataSource = new SimpleDriverDataSource();             
    dataSource.setDriver(new org.postgresql.Driver());
    dataSource.setUrl("jdbc:postgresql://localhost:5432/batch");
    dataSource.setUsername("batch");
    dataSource.setPassword("batch");                
    return dataSource;
}

//Oracle Data source for source

@Primary
@Bean(name="oracleDBDataSource")     
public DataSource oracleDBDataSource() throws SQLException {

    final SimpleDriverDataSource dataSource = new SimpleDriverDataSource();        
    dataSource.setDriver(new oracle.jdbc.driver.OracleDriver());  //(new oracle.jdbc.pool.OracleDataSource());
    dataSource.setUrl("jdbc:oracle:thin:@xxxxxxx.com:1521/xxxxx");
    dataSource.setUsername("xxxx");
    dataSource.setPassword("xxxxx");
    return dataSource;
}

@Bean
public JdbcTemplate jdbcTemplate(final DataSource dataSource) {

    return new JdbcTemplate(dataSource);
}


@Bean
public JobExecutionListener listener() {

    return new JobCompletionNotificationListener(new JdbcTemplate(dataSource2));
}

// end::listener[]

// tag::jobstep[]
@Bean
public Job importUserJob() {

    return jobBuilderFactory.get("importUserJob")
            .incrementer(new RunIdIncrementer())
            .listener(listener())
            .flow(step1())
            .end()
            .build();
}



@Bean
public Step step1() {

    return stepBuilderFactory.get("step1")
            .<Event, Event> chunk(10)
            .reader(reader())
            .processor(processor())
            .writer(writer())
            .build();
}
// end::jobstep[]

}

1 Answers1

0

Your question is very broad and shows very little research effort. Nevertheless, it is possible to configure a database reader and a database writer in a spring batch job.

You will have to configure two datasources (the source datasource being your Teradata DB and the target datasource being your Postgres DB). See this question and answer for information how to do this using Spring Batch: Use of multiple DataSources in Spring Batch

Then you have to create a reader with the source datasource and a writer with the target datasource.

Slightly adjusted the standard Spring Batch Boot tutorial to your situation:

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Autowired
    @Qualifier("source")
    public DataSource source;

    @Autowired
    @Qualifier("target")
    public DataSource target;

    @Bean
    public ItemReader<Person> reader() {
        JdbcCursorItemReader<Person> reader = new JdbcCursorItemReader<Person>();
        String sql = "select * from Person";
        reader.setSql(sql);
        reader.setDataSource(source);
        reader.setRowMapper(rowMapper());
        return reader;
    }    

    @Bean
    public JdbcBatchItemWriter<Person> writer() {
        JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<Person>();
        writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
        writer.setSql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)");
        writer.setDataSource(target);
        return writer;
    }        

    @Bean
    public Job yourJob() {
        return jobBuilderFactory.get("yourJob")
                .incrementer(new RunIdIncrementer())                    
                .flow(step1())
                .end()
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<Person, Person> chunk(10)
                .reader(reader())                    
                .writer(writer())
                .build();
    }      
} 

For the rest of the configuration of this example please checkout https://spring.io/guides/gs/batch-processing/ and for more detailed explanation checkout this tutorial http://www.codingpedia.org/ama/spring-batch-tutorial-with-spring-boot-and-java-configuration/ (you can find an example of a RowMapper there as well)

Community
  • 1
  • 1
Sander_M
  • 1,109
  • 2
  • 18
  • 36
  • Thanks for the response. I was more worried about connecting to Teradata DB server from the spring batch(dependency etc.) and having multiple data sources. Before starting with Teradata DB, i was trying to make the multiple data source job work with an Oracle DB as source & Postgres DB as target. I followed different suggestions mentioned for multi-source spring batch scenarios. Still i am getting errors. Will update the code & errors. Please let me know if you see anything wrong with it. – pstimeforj Aug 12 '16 at 06:45