Spring Batch with Spring Cloud AWS S3 returns 301 for EU bucket

Question

I'm attempting to read a file in Spring Batch that is hosted on S3. I want to access the file directly from S3, not download it as a local file and then read.

Here's my reader:

class MyReader extends FlatFileItemReader<MyReadItem> {

    @Value('${com.me.s3bucket}')
    private String s3bucket;

    MyReader (DefaultLineMapper< MyReadItem > myLineMapper, ResourceLoader resourceLoader) {
        this.linesToSkip = 1
        this.lineMapper = myLineMapper
        this.resource = resourceLoader.getResource("s3://${s3bucket}/path/to/myfile.csv")
    }
}

The reader is wired up in my batch config file:

@Bean
public ItemReader< MyReadItem > reader() {
    return new MyReader(myLineMapper(), resourceLoader)
}

@Autowired
ResourceLoader resourceLoader

// myLineMapper ...

When I run the batch job, I get a 301 exception:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Moved Permanently (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: 1DAB29F45F62E0D9)

S3 301 exceptions appear to be because the region isn't defined when the request is made (example here), but my application.yaml file is shown below where I define the region:

cloud:
    aws:
        region:
            static: eu-west-1
            auto: false

What's going on here? Is the problem something to do with the order the beans are wired up, perhaps before the configuration file is read?

To test, I have used this method to download S3 as a temp file within the same application and it works ok:

@Value('${com.me.s3bucket}')
private String s3bucket;

public static final String PREFIX = "s3temp";
public static final String SUFFIX = ".tmp";

public File downloadS3File(String resourcePath) throws IOException {
    Resource resource = this.resourceLoader.getResource("s3://${s3bucket}/${resourcePath}");

    InputStream inputStream = resource.getInputStream();

    final File tempFile = File.createTempFile(PREFIX, SUFFIX);
    tempFile.deleteOnExit();
    try {
        FileOutputStream out = new FileOutputStream(tempFile)
        IOUtils.copy(inputStream, out);
    }
    finally {
    }

    return tempFile;
}

Is there any way of explicitly specifying the region when using resource loader?

score 1 · Accepted Answer · answered Jul 21 '15 at 10:57

OK, so the problem was with the fact that the bucket wasn't being set and was null.

@Value('${com.me.s3bucket}')
private String s3bucket;

I don't know why this was, but when I moved this out of the class and sent the value as a job parameter, it worked.

Spring Batch with Spring Cloud AWS S3 returns 301 for EU bucket

1 Answers1