The FlatFileItemReader read only one line from the CSV file - Spring Batch

Question

I'm creating a Spring Batch Job to populate Data into a Database table from a given CSV file. I created a customized FlatFileItemReader. my problem is that the read() method is invoked only one time so only the first line of my CSV file is inserted into the database.

@Configuration
@EnableBatchProcessing 
public class SpringBatchConfig {

private MultipartFile[] files;

@Bean
public Job job(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory,
        ItemReader<MyModelEntity> itemReader,
        ItemWriter<MyModelEntity> itemWriter) {

    Step step = stepBuilderFactory.get("Load-CSV-file_STP")
            .<MyModelEntity, MyModelEntity > chunk(12)
            .reader(itemReader)
            .writer(itemWriter).build();

    return jobBuilderFactory.get("Load-CSV-Files").
            incrementer(new RunIdIncrementer()) /
            .start(step) 
            .build();
}
 @Bean
  ItemReader<MyModelEntity> myModelCsvReader() throws Exception {
        return new MyModelCsvReader();
    }

 }

the myModelCsvReader

@Component
@StepScope
public class MyModelCsvReader implements   ItemReader<MyModelEntity>{


@Value("#{jobParameters['SDH']}")
private String sdhPath;

private  boolean batchJobState= false;

@Autowired 
MyModelFieldSetMapper myModelFieldSetMapper;

public LineMapper<MyModelEntity> lineMapper() throws Exception {
    DefaultLineMapper<MyModelEntity> defaultLineMapper = new 
    DefaultLineMapper<MyModelEntity>();
    DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();

    lineTokenizer.setDelimiter(",");
    lineTokenizer.setStrict(false);  
    lineTokenizer.setNames(new String[] 
      {
      "clientId","ddId","institName","progName",
      "qual","startDate","endDate","eType", "country","comments"
      });

     defaultLineMapper.setLineTokenizer(lineTokenizer);
     defaultLineMapper.setFieldSetMapper(myModelFieldSetMapper);
    return defaultLineMapper;}

@Override
public   MyModelEntity  read()
        throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
     //if(!batchJobState )
    {
    FlatFileItemReader<MyModelEntity> flatFileItemReader = new 
    FlatFileItemReader<MyModelEntity>();
    flatFileItemReader.setMaxItemCount(2000);
    flatFileItemReader.setResource(new UrlResource("file:\\"+sdhPath));
    flatFileItemReader.setName("CSV-Reader");
    flatFileItemReader.setLinesToSkip(1);
    flatFileItemReader.setLineMapper(lineMapper());
    flatFileItemReader.open(new ExecutionContext());         
    batchJobState=true; 
    return flatFileItemReader.read();

    }
    // return null;
}
}

the FieldSetMapper Implementation

 @Component
public class MyModelFieldSetMapper  implements FieldSetMapper<MyModelEntity>  {

//private SiteService siteService =BeanUtil.getBean(SiteServiceImpl.class);

 @Autowired 
 private SiteService siteService;

@Override
public MyModelEntity mapFieldSet(FieldSet fieldSet ) throws BindException {
    if(fieldSet == null){
           return null;
       }
    MyModelEntity educationHistory = new MyModelEntity();
    // setting MyModelAttributes Values 
    return myModel; 
}
}

any conribution is welcomed . thanks // thereader after extending FlatFileItemReader

@Component
@StepScope
public class CustomUserItemReader  extends FlatFileItemReader<User> {
@Value("#{jobParameters['UserCSVPath']}")
private String UserCSVPath;
private boolean batchJobState;

public   CustomUserItemReader() throws Exception {
    super();

    setResource(new UrlResource("file:\\"+UserCSVPath));
    setLineMapper(lineMapper());
    afterPropertiesSet();
    setStrict(false);

}

 public LineMapper<User> lineMapper() throws Exception {
 DefaultLineMapper<User> defaultLineMapper = 
    new DefaultLineMapper<User>();
    DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();

    lineTokenizer.setDelimiter(",");
    lineTokenizer.setStrict(false);
    lineTokenizer.setNames(new String[]{"name", "dept", 
   "salary","endDate"});
    defaultLineMapper.setLineTokenizer(lineTokenizer);
    defaultLineMapper.setFieldSetMapper(new CustomUserFieldSetMapper());
    //defaultLineMapper.setFieldSetMapper(fieldSetMapper);
    return defaultLineMapper;}
 @Override
 public   User  read()
        throws Exception, UnexpectedInputException, ParseException, 
   NonTransientResourceException {
    //if(!batchJobState )
    {
    flatFileItemReader).size())
      //  flatFileItemReader.setMaxItemCount(2000);
        this.setResource(new UrlResource("file:\\"+UserCSVPath));
        this.setName("CSV-Reader");
        this.setLinesToSkip(1);
        //flatFileItemReader.setLineMapper(lineMapper());
        this.open(new ExecutionContext());
        User e = this.read();
        batchJobState = true;
        return e ;

    }
    // return null;
}

public
String getUserCSVPath() {
    return UserCSVPath;
}

public
void setUserCSVPath(String userCSVPath) {
    UserCSVPath = userCSVPath;
}
}

I can see when the read method is called you are creating a flatFileItemReader for every row. instead of this you can extend FlatFileItemReader instead of implementing ItemReader and use `super.read();` You can override lineMapper and fieldSetMapper — Sujay Mohan, Oct 13 '19 at 15:05
Hi, @SujayMohan thanks for your message; I tried that before but it gives me an error "Illegal ARguments LineMapper is Required "I think you are right I have to Override the Linemapper. Could you please share with me an implementation. thank you — BSL, Oct 13 '19 at 16:48
Reader implementation is wrong. Try to override the FlatFileItemReader and do the necessary change you want to do. Here you have a reader and you are returning instance of FlatFileItemReader instance every record it reads. this is wrong. — Sushil Behera, Oct 13 '19 at 19:46
@SushilBehera, Thanks for your message, please consider that im beginner it's my first experience with Spring Batch, according to SujayMohan I have removed the read() method and set all the properties of my Reader (which extends FlatFileItemReader) within the constructor but I m facing pb on how and where I should send my List of Models to be persisted by the Writer in the DB. please any links or examples are welcomed. — BSL, Oct 13 '19 at 20:00
Is it possible for you to share the code. I can take a look. Basically your readers read method need to return only one entity every time. When you are out of any record then return null. Spring batch calls the read method in loop until you return null. It will call the writers write method when the commit-interval meet to write the list to DB. — Sushil Behera, Oct 13 '19 at 20:03
This Code is the same as the previous one. I created a small separate project which i added in the question — BSL, Oct 13 '19 at 21:01

score 2 · Answer 1 · answered Oct 13 '19 at 21:49

Thanks for all your suggestions, even if i have implemented ItemReader<>. I fixed the problem by moving the instantiation of the FlatFileItemReader Out from the read() method. that was creating a new FlatFileItemReader in each loop and reading only the first line for each object created . thanks

The FlatFileItemReader read only one line from the CSV file - Spring Batch

1 Answers1