I am writing a Spring Batch Application which needs to read a file from an AWS S3 bucket.
This is my AWS Config Java Class,
@Configuration
public class AWSConfig{
@Value("${cloud.aws.credentials.accessKey}")
private String accessKey;
@Value("${cloud.aws.credentials.secretKey}")
private String secretKey;
@Value("${cloud.aws.region}")
private String region;
@Bean
public BasicAWSCredentials basicAWSCredentials() {
return new BasicAWSCredentials(accessKey, secretKey);
}
@Bean
public AmazonS3Client amazonS3Client(AWSCredentials awsCredentials) {
AmazonS3Client amazonS3Client = (AmazonS3Client) AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
.withRegion(region)
.build();
return amazonS3Client;
}
}
This is my aws-context.xml(located in the resources/) file which is to modifiy the default ResourceLoader,
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:aws-context="http://www.springframework.org/schema/cloud/aws/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/cloud/aws/context
http://www.springframework.org/schema/cloud/aws/context/spring-cloud-aws-context.xsd">
<aws-context:context-resource-loader amazon-s3="amazonS3Client" />
</beans>
This is my SpringBatchConfig.java class,
@Configuration
@EnableBatchProcessing
public class SpringBatchConfig {
@Autowired
private ResourceLoader resourceLoader;
@Bean
public Job job(JobBuilderFactory jobBuilderFactory,
StepBuilderFactory stepBuilderFactory,
ItemReader<User> itemReader,
ItemProcessor<User, User> itemProcessor,
ItemWriter<User> itemWriter
) {
Step step = stepBuilderFactory.get("ETL-file-load")
.<User, User>chunk(100)
.reader(itemReader)
.processor(itemProcessor)
.writer(itemWriter)
.build();
return jobBuilderFactory.get("ETL-Load")
.incrementer(new RunIdIncrementer())
.start(step)
.build();
}
@Bean
public FlatFileItemReader<User> itemReader() throws IOException {
FlatFileItemReader<User> flatFileItemReader = new FlatFileItemReader<>();
flatFileItemReader.setResource(resourceLoader.getResource("s3://" + "<bucket-name>" + "/" + "<key>"));
flatFileItemReader.setName("CSV-Reader");
flatFileItemReader.setLinesToSkip(1);
flatFileItemReader.setLineMapper(lineMapper());
return flatFileItemReader;
}
@Bean
public LineMapper<User> lineMapper() {
DefaultLineMapper<User> defaultLineMapper = new DefaultLineMapper<>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[]{"id", "name", "dept", "salary"});
BeanWrapperFieldSetMapper<User> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
fieldSetMapper.setTargetType(User.class);
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(fieldSetMapper);
return defaultLineMapper;
}
}
I have followed the answer given by @mtoutcalt in this StackOverflow thread to configure this, Spring Batch - Read files from Aws S3
And also this documentation: https://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_resource_handling
Issues I face,
1) In the SpringBatchConfig.java it when try to autowire ResourceLoader it says (I'm using IntelliJIdea),
Could not autowire. There is more than one bean of 'ResourceLoader' type.
Beans:
(aws-config.xml) webApplicationContext (Spring Web)
2) When I run the Batch Application it says,
Caused by: java.lang.IllegalStateException: Input resource must exist (reader is in 'strict' mode): ServletContext resource [/s3://<bucket-name>/<key>]
Can someone please help with fixing this?
Regards