Spring Batch "Invalid object name BATCH_JOB_INSTANCE"

Question

I've created a spring batch to query a Azure SQL server database and write the data into a CSV file. I do not have create permissions for the database. I get this error Invalid Object name BATCH_JOB_INSTANCE on running the batch. I don't want the spring batch meta-data tables to be created in the main database. Or it would be helpful if I can have them in another local or in-memory db like h2db. I've also added spring-batch-initialize-schema=never already, which was the case with most answers to similar questions on here, but that didn't help.

Edit:

I resolved the Invalid Object name error by preventing the metadata tables from being created into the main database by extending the DefaultBatchConfigurer Class and Overriding the setDataSource method, thus having them created in the in-memory map-repository. Now I want to try two options:

1. How to have the meta data tables to be created in a local db or in-memory db like h2db.
2. Or If I have the meta data tables created already in the main database, in a different schema than my main table I'm fetching from. How to point my job to those meta-data tables in another schema, to store the job and step details data in those.

@Configuration
public class SpringBatchConfig extends DefaultBatchConfigurer{

@Override
    public void setDataSource(DataSource datasource) {

    }
...

My application.properties file looks like this:

spring.datasource.url=
spring.datasource.username=
spring.datasource.password=
spring.datasource.driver-class-name=com.microsoft.sqlserver.jdbc.SQLServerDriver

spring-batch-initialize-schema=never
spring.batch.job.enabled=false

spring.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.SQLServer2012Dialect

Answer 1

I've created a demo with two datasources. Batch metadata will sotre in H2 DB and the Job datasource is Azure SQL.

Here is the project structure:

1. We need define a DataSourceConfig class and use @Primary annotation for DataSource bean:

@Configuration
public class DataSourceConfig {

    @Bean(name = "mssqlDataSource")
    @ConfigurationProperties(prefix = "spring.datasource")
    public DataSource appDataSource(){
        return DataSourceBuilder.create().build();
    }
    
    @Bean(name = "h2DataSource")
    @Primary
    // @ConfigurationProperties(prefix="spring.datasource.h2")
    public DataSource h2DataSource() {
        return DataSourceBuilder.create()
                .url("jdbc:h2:mem:thing:H2;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE")
                .driverClassName("org.h2.Driver")
                .username("sa")
                .password("")                   
                .build();
    }
    
}

2. In the ItemReaderDbDemo class, we use @Autowired @Qualifier("mssqlDataSource") to specify the dataSource in the Spring Batch task:

@Configuration
public class ItemReaderDbDemo {
    
    //generate task Object
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    
    //Step exec tasks
    //generate step Object
    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    @Qualifier("mssqlDataSource")
    private DataSource dataSource;

    @Autowired
    @Qualifier("dbJdbcWriter")
    private ItemWriter<? super Todo> dbJdbcWriter;
    
    @Bean
    public Job itemReaderDbDemoJob() {
        return jobBuilderFactory.get("itemReaderDbDemoJob").start(itemReaderDbStep()).build();
    }

    @Bean
    public Step itemReaderDbStep() {
        return stepBuilderFactory.get("itemReaderDbStep")
                .<Todo,Todo>chunk(2)
                .reader(dbJdbcReader())
                .writer(dbJdbcWriter)
                .build();
    }

    @Bean
    @StepScope
    public JdbcPagingItemReader<Todo> dbJdbcReader() {
        JdbcPagingItemReader<Todo> reader = new JdbcPagingItemReader<Todo>();
        reader.setDataSource(dataSource);
        reader.setFetchSize(2);
        reader.setRowMapper(new  RowMapper<Todo>() {
            @Override
            public Todo mapRow(ResultSet rs, int rowNum) throws SQLException {
                Todo todo = new Todo();
                todo.setId(rs.getLong(1));
                todo.setDescription(rs.getString(2));
                todo.setDetails(rs.getString(3));
                return todo;
            }   
            
        });
        SqlServerPagingQueryProvider provider = new  SqlServerPagingQueryProvider();
        provider.setSelectClause("id,description,details");
        provider.setFromClause("from dbo.todo");
        
        //sort
        Map<String,Order> sort = new HashMap<>(1);
        sort.put("id", Order.DESCENDING);
        provider.setSortKeys(sort);
        
        reader.setQueryProvider(provider);
        return reader;
    }
}

3. Here is my application.properties:

logging.level.org.springframework.jdbc.core=DEBUG

spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.datasource.jdbcUrl=jdbc:sqlserver://josephserver2.database.windows.net:1433;database=<Your-Database-Name>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;
spring.datasource.username=<Your-UserName>
spring.datasource.password=<Your-Password>
spring.datasource.initialization-mode=always

4. It return expected result from my Azure SQL. By the way, my Azure sql username does not have create permissions for the database.
   The result shows:

Answer 2

How to have the meta data tables to be created in a local db or in-memory db like h2db.

You can use spring.batch.initialize-schema=embedded for that.

Or If I have the meta data tables created already in the main database, in a different schema than my main table I'm fetching from. How to point my job to those meta-data tables in another schema, to store the job and step details data in those.

spring batch works against a datasource, not a particular schema. If meta-data tables are in a different schema, then you need to create a second datasource pointing to that schema and set it on the job repository.

Answer 3

I know this post is a little bit old, but I'd like to give an update. For newer versions of Spring Boot spring.batch.initialize-schema is deprecated. I'm using Spring Boot 2.7.1 and the newer property is spring.batch.jdbc.initialize-schema. In my case: when I was receiving the error message was due that the user did not have the CREATE TABLE permission to create the corresponding spring bacth tables. Adding the permissions fix the issue.