I have to write a batch job which will have as its source, a Teradata DB & as target, a Postgres DB using Spring batch framework. Could anyone let me know or refer me to a tutorial on how to achieve this, if at all its possible?
If not, what are the other options I could explore in order to do this?
UPDATE
I have a spring batch job, trying to make it work with Oracle(Source) & Postgres(Target), later I will be changing the source to Teradata. However, I am getting below error, which i assume is due to, the fact that job sequences are being looked up in the Oracle DB(where the sequences cant be created, as the DB user has Read Only access). How do I make the Job, create the sequences in the target DB(Postgres)?
Errors:
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:227)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:121)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:115)
at org.springframework.boot.SpringApplication.runCommandLineRunners(SpringApplication.java:672)
... 5 common frames omitted
Caused by: java.sql.SQLException: ORA-02289: sequence does not exist
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:745)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:210)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:804)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1049)
at oracle.jdbc.driver.T4CStatement.executeMaybeDescribe(T4CStatement.java:845)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1154)
at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1313)
at org.springframework.jdbc.support.incrementer.AbstractSequenceMaxValueIncrementer.getNextKey(AbstractSequenceMaxValueIncrementer.java:69)
... 40 common frames omitted
2016-08-12 12:34:57.467 INFO 11660 --- [main] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@1cea91 b: startup date [Fri Aug 12 12:34:22 IST 2016]; root of context hierarchy
2016-08-12 12:34:57.469 INFO 11660 --- [main] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
** BatchConfiguration.java **
@Configuration
@EnableBatchProcessing
@ComponentScan(basePackageClasses = MyBatchConfigurer.class)
public class BatchConfiguration {
@Autowired
public JobBuilderFactory jobBuilderFactory;
@Autowired
public StepBuilderFactory stepBuilderFactory;
@Autowired
@Qualifier("oracleDBDataSource")
public DataSource dataSource1;
@Autowired
@Qualifier("postgresDataSource")
public DataSource dataSource2;
//Read an Event from oracle DB
@Bean
public ItemReader<Event> reader(){
JdbcCursorItemReader<Event> databaseReader = new JdbcCursorItemReader<>();
String sqlQ = "select * from test_event";
databaseReader.setDataSource(dataSource1);
databaseReader.setSql(sqlQ);
databaseReader.setRowMapper(new BeanPropertyRowMapper<>(Event.class));
return databaseReader;
}
//EventItemProcessor
@Bean
public EventItemProcessor processor() {
return new EventItemProcessor();
}
@Bean
public JdbcBatchItemWriter<Event> writer() {
JdbcBatchItemWriter<Event> writer = new JdbcBatchItemWriter<Event>();
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Event>());
writer.setSql("insert into EVENTS (event_id, event_title, event_start_date, event_end_date, event_type,need_offline_sync, event_status, created_by, created_date, last_updated_by,last_updated_date) VALUES (:event_id, :event_title, :event_start_date, :event_end_date, :event_type,:need_offline_sync, :event_status, :created_by, :created_date, :last_updated_by, :last_updated_date)");
writer.setDataSource(dataSource2);
return writer;
}
//Postgres Data source for Target
@Bean(name="postgresDataSource")
public DataSource postgresDataSource() throws SQLException {
final SimpleDriverDataSource dataSource = new SimpleDriverDataSource();
dataSource.setDriver(new org.postgresql.Driver());
dataSource.setUrl("jdbc:postgresql://localhost:5432/batch");
dataSource.setUsername("batch");
dataSource.setPassword("batch");
return dataSource;
}
//Oracle Data source for source
@Primary
@Bean(name="oracleDBDataSource")
public DataSource oracleDBDataSource() throws SQLException {
final SimpleDriverDataSource dataSource = new SimpleDriverDataSource();
dataSource.setDriver(new oracle.jdbc.driver.OracleDriver()); //(new oracle.jdbc.pool.OracleDataSource());
dataSource.setUrl("jdbc:oracle:thin:@xxxxxxx.com:1521/xxxxx");
dataSource.setUsername("xxxx");
dataSource.setPassword("xxxxx");
return dataSource;
}
@Bean
public JdbcTemplate jdbcTemplate(final DataSource dataSource) {
return new JdbcTemplate(dataSource);
}
@Bean
public JobExecutionListener listener() {
return new JobCompletionNotificationListener(new JdbcTemplate(dataSource2));
}
// end::listener[]
// tag::jobstep[]
@Bean
public Job importUserJob() {
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener())
.flow(step1())
.end()
.build();
}
@Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<Event, Event> chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
// end::jobstep[]
}