4

I am writing a Spring Batch Application which needs to read a file from an AWS S3 bucket.

This is my AWS Config Java Class,

@Configuration
public class AWSConfig{

    @Value("${cloud.aws.credentials.accessKey}")
    private String accessKey;

    @Value("${cloud.aws.credentials.secretKey}")
    private String secretKey;

    @Value("${cloud.aws.region}")
    private String region;

    @Bean
    public BasicAWSCredentials basicAWSCredentials() {
        return new BasicAWSCredentials(accessKey, secretKey);
    }

    @Bean
    public AmazonS3Client amazonS3Client(AWSCredentials awsCredentials) {
        AmazonS3Client amazonS3Client = (AmazonS3Client) AmazonS3ClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
                .withRegion(region)
                .build();
        return amazonS3Client;
    }

}

This is my aws-context.xml(located in the resources/) file which is to modifiy the default ResourceLoader,

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:aws-context="http://www.springframework.org/schema/cloud/aws/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/cloud/aws/context
        http://www.springframework.org/schema/cloud/aws/context/spring-cloud-aws-context.xsd">

    <aws-context:context-resource-loader amazon-s3="amazonS3Client" />

</beans>

This is my SpringBatchConfig.java class,

@Configuration
@EnableBatchProcessing
public class SpringBatchConfig {

    @Autowired
    private ResourceLoader resourceLoader;

    @Bean
    public Job job(JobBuilderFactory jobBuilderFactory,
                   StepBuilderFactory stepBuilderFactory,
                   ItemReader<User> itemReader,
                   ItemProcessor<User, User> itemProcessor,
                   ItemWriter<User> itemWriter
    ) {

        Step step = stepBuilderFactory.get("ETL-file-load")
                .<User, User>chunk(100)
                .reader(itemReader)
                .processor(itemProcessor)
                .writer(itemWriter)
                .build();


        return jobBuilderFactory.get("ETL-Load")
                .incrementer(new RunIdIncrementer())
                .start(step)
                .build();
    }

    @Bean
    public FlatFileItemReader<User> itemReader() throws IOException {

        FlatFileItemReader<User> flatFileItemReader = new FlatFileItemReader<>();
        flatFileItemReader.setResource(resourceLoader.getResource("s3://" + "<bucket-name>" + "/" + "<key>"));
        flatFileItemReader.setName("CSV-Reader");
        flatFileItemReader.setLinesToSkip(1);
        flatFileItemReader.setLineMapper(lineMapper());
        return flatFileItemReader;
    }

    @Bean
    public LineMapper<User> lineMapper() {

        DefaultLineMapper<User> defaultLineMapper = new DefaultLineMapper<>();
        DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();

        lineTokenizer.setDelimiter(",");
        lineTokenizer.setStrict(false);
        lineTokenizer.setNames(new String[]{"id", "name", "dept", "salary"});

        BeanWrapperFieldSetMapper<User> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
        fieldSetMapper.setTargetType(User.class);

        defaultLineMapper.setLineTokenizer(lineTokenizer);
        defaultLineMapper.setFieldSetMapper(fieldSetMapper);

        return defaultLineMapper;
    }

}

I have followed the answer given by @mtoutcalt in this StackOverflow thread to configure this, Spring Batch - Read files from Aws S3

And also this documentation: https://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_resource_handling

Issues I face,

1) In the SpringBatchConfig.java it when try to autowire ResourceLoader it says (I'm using IntelliJIdea),

Could not autowire. There is more than one bean of 'ResourceLoader' type.
Beans:
    (aws-config.xml) webApplicationContext   (Spring Web) 

2) When I run the Batch Application it says,

Caused by: java.lang.IllegalStateException: Input resource must exist (reader is in 'strict' mode): ServletContext resource [/s3://<bucket-name>/<key>]

Can someone please help with fixing this?

Regards

samme4life
  • 1,143
  • 2
  • 14
  • 20
  • Did you find the solution for this issue? – Jeff Cook Jan 06 '21 at 05:55
  • The first error message "There is more than one bean of 'ResourceLoader' type" could be caused because you are using either xml amazonS3Client bean, and Spring web AmazonS3Client bean as well. There is a conflict between the beans, use only one or define the proper one with @Qualifier. – danyPasillas Mar 04 '21 at 16:20
  • Did you find the solution for the 2nd exception as I am getting that exception but s3 file exist as s://mybucket/directory/myfile.csv – Ashish Awasthi Aug 21 '22 at 19:54

0 Answers0