I need to create a process that downloads a file from an API endpoint and uploads it to another API endpoint. The file has a maximum size of 100MB but we will have many processes running in parallel. I'm trying to achieve it with Spring WebClient and without storing the files in memory. The current code is storing the file in memory because the tests with big files are throwing OutOfMemoryError.
Copy method
public Mono<SyncStatusType> copy(Strin path1, String path2) {
return this.download(path1)
.collectList()
.flatMap(dataBuffers ->
this.upload(path2, Mono.just(dataBuffers).flatMapMany(Flux::fromIterable)))
.map(item -> SyncStatusType.SUCCESSFUL);
}
Download method
private Flux<DataBuffer> download(String path) {
return this.getWebClient()
.get()
.uri(uriBuilder -> uriBuilder
.path(path)
.build())
.retrieve()
.bodyToFlux(DataBuffer.class);
}
Upload method
private Mono<CustomFileDto> upload(String path, Flux<DataBuffer> dataBuffer) {
return this.getWebClient()
.put()
.uri(uriBuilder -> uriBuilder
.path(path)
.build())
.contentType(MediaType.APPLICATION_JSON)
.body(BodyInserters.fromDataBuffers(dataBuffer))
.retrieve()
.bodyToMono(CustomFileDto.class)
}
How can I modify this code so that it doesn't store the file content in memory?
