0

I'm working on process which uses Spring Integration and Spring Batch

1)Using Spring integration I will poll remote sftp dir to get different csv files as Message

2)Message which carries csv file as payload is sent downstream to Transformer which will transform Message to JobLaunchRequest

3)Spring batch reads csv files and dumps into DB

Question: For each csv file I need to configure (ItemReader, ItemWriter, Step, Job) So with that into consideration if I have to deal with 10 different csv files do I have to configure all 4 beans listed above for each csv? CSVs differs in HeaderNames and HeaderCount and each csv has different JPA Entity

Eventually I will have 40 @Bean Configurations which ideally I think is bad Can anyone suggest me if this is how spring batch is made to work or there is other way to make it one common dynamic bean for different CSVs

Here is code:

IntegartionFlow:

 @Bean
public IntegrationFlow integrationFlow(JobLaunchingGateway jobLaunchingGateway) {
    return IntegrationFlows.from(Sftp.inboundAdapter(sftpSessionFactory)
                    .remoteDirectory("/uploads")
                    .localDirectory(new File("C:\\Users\\DELL\\Desktop\\local"))
                    .patternFilter("*.csv")
                    .autoCreateLocalDirectory(true)
            , c -> c.poller(Pollers.fixedRate(1000).taskExecutor(taskExecutor()).maxMessagesPerPoll(1)))
            .transform(fileMessageToJobRequest())
            .handle(jobLaunchingGateway)
            .log(LoggingHandler.Level.WARN, "headers.id + ': ' + payload")
            .route(JobExecution.class, j -> j.getStatus().isUnsuccessful() ? "jobFailedChannel" : "jobSuccessfulChannel")
            .get();
}

Transformer:

 @Transformer
public JobLaunchRequest toRequest(Message<File> message) {
    JobParametersBuilder jobParametersBuilder =
            new JobParametersBuilder();

    jobParametersBuilder.addString(fileParameterName,
            message.getPayload().getAbsolutePath());
    jobParametersBuilder.addLong("key.id", System.currentTimeMillis());

    return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}

Batch Job:

 @Bean
public Job vendorMasterBatchJob(Step vendorMasterStep) {
    return jobBuilderFactory.get("vendorMasterBatchJob")
            .incrementer(new RunIdIncrementer())
            .start(vendorMasterStep)
            .listener(deleteInputFileJobListener)
            .build();
}

Batch Step:

  @Bean
public Step vendorMasterStep(FlatFileItemReader<ERPVendorMaster> vendorMasterReader,
                             JpaItemWriter<ERPVendorMaster> vendorMasterWriter) {
    return stepBuilderFactory.get("vendorMasterStep")
            .<ERPVendorMaster, ERPVendorMaster>chunk(chunkSize)
            .reader(vendorMasterReader)
            .writer(vendorMasterWriter)
            .faultTolerant()
            .skipLimit(Integer.MAX_VALUE)
            .skip(RuntimeException.class)
            .listener(skipListener)
            .build();
}

ItemWriter:

 @Bean
public JpaItemWriter<ERPVendorMaster> vendorMasterWriter() {
    return new JpaItemWriterBuilder<ERPVendorMaster>()
            .entityManagerFactory(entityManagerFactory)
            .build();
}

ItemReader:

 @Bean
@StepScope
public FlatFileItemReader<ERPVendorMaster> vendorMasterReader(@Value("#{jobParameters['input.file.name']}") String fileName) {
    return new FlatFileItemReaderBuilder<ERPVendorMaster>()
            .name("vendorMasterItemReader")
            .resource(new FileSystemResource(fileName))
            .linesToSkip(1)
            .delimited()
            .names(commaSeparatedVendorMasterHeaderValues.split(","))
            .fieldSetMapper(new BeanWrapperFieldSetMapper<ERPVendorMaster>() {{
                setConversionService(stringToDateConversionService());
                setTargetType(ERPVendorMaster.class);
            }})
            .build();
}

I'm very new to Spring boot any help will be appreciated

Thanks

abdur rehman
  • 55
  • 8
  • Does this answer your question? [Spring Batch reader using generics](https://stackoverflow.com/questions/33627388/spring-batch-reader-using-generics) – Mahmoud Ben Hassine Jan 08 '21 at 09:55
  • @MahmoudBenHassine Not really.. however I got some idea from [this](https://stackoverflow.com/questions/59548732/how-to-create-a-generic-flatfileitemreader-to-read-csv-files-with-different-head?noredirect=1&lq=1) thread to how to initialize reader, but still that is not what I need – abdur rehman Jan 08 '21 at 14:44
  • Have you tried the following: create a base type for your classes that is used in an abstract generic configuration class + one concrete configuration subclass for each type? – Mahmoud Ben Hassine Jan 11 '21 at 08:03
  • @MahmoudBenHassine Can you please provide a simple code sample? it would be more helpful than trying to understand it in one single line comment.. Thanks alot – abdur rehman Jan 11 '21 at 09:32
  • Well, I wanted to know if you tried this approach before posting an answer. This is not related to Spring Batch per se, it's a matter of configuring generic Spring beans for different types. I don't have an example but here is some guidance on the idea: 1) create a base type for all your types, like the `GenericFilePOJO` in the link you shared. 2) create a base configuration class `class AbstractConfig` where you define common config and 3) create a concrete config class for each of your sub types, for example: `class ERPVendorMasterConfig extends AbstractConfig`. – Mahmoud Ben Hassine Jan 11 '21 at 10:28

0 Answers0