Stream upload 'POST' in Spring WebClient

Question

I am uploading streams of (raw bytes) data using HTTP posts using WebClient:

    final byte[] rawData = IOUtils.toByteArray(sourceInputStream);
    webClient.post()
             .uri(uri)
             .contentType(MediaType.APPLICATION_OCTET_STREAM)
             .bodyValue(rawData)
             .exchange()...

I am concerned there is a potentially a lot of memory used given sometimes these objects can be quite big (~200Mb) so would like to read directly from the InputStream and upload as a stream.

I tried:

 bodyValue(BodyInserters.fromResource(new InputStreamResource(inputStream)))

but got exception Content type 'application/octet-stream' not supported for bodyType=org.springframework.web.reactive.function.BodyInserters

So I then tried removing the header but the data is then corrupted.

Is there a way to stream the data without passing through the in memory 'buffer' rawData[]?

Thanks

My solution below - if you can update it with taking chunks of bytes from an input stream then it would be an improvement. — user1016765, Jul 21 '21 at 21:51

score 4 · Accepted Answer · answered Nov 17 '21 at 16:46

Your first try was almost correct, however you need to use body(...) instead of bodyValue(...):

body(BodyInserters.fromResource(new InputStreamResource(inputStream)))

This is because bodyValue(...) wraps your resource inserter in a value inserter, which will then try to serialize the resource inserter itself and fail with the error you received.

score 0 · Answer 2 · answered Jul 21 '21 at 21:43

I ended up keeping the rawData[] buffer and specifying contentLength in the request:

webClient.post()
         .uri(uri)
         .contentType(MediaType.APPLICATION_OCTET_STREAM)
         .contentLength(bytes.length)
         .bodyValue(bytes)
         .exchange()
         .block(Duration.ofSeconds(30));

For very large files I did chunked uploads - for example 1Mb chunks of a file might look like:

POST: https://sharepoint.acme.com:443/sharepoint-site/_api/web/GetFileByServerRelativeUrl(@f)/StartUpload(uploadId=guid'680f93eb-8c6a-443d-87ad-1e1b324ea678')?@f='/sharepoint-site/folderA/subFolderB/etc/example.file'
POST: https://sharepoint.acme.com:443/sharepoint-site/_api/web/GetFileByServerRelativeUrl(@f)/ContinueUpload(uploadId=guid'680f93eb-8c6a-443d-87ad-1e1b324ea678',fileOffset=1048576)?@f='/sharepoint-site/folderA/subFolderB/etc/example.file'
POST: https://sharepoint.acme.com:443/sharepoint-site/_api/web/GetFileByServerRelativeUrl(@f)/ContinueUpload(uploadId=guid'680f93eb-8c6a-443d-87ad-1e1b324ea678',fileOffset=2097152)?@f='/sharepoint-site/folderA/subFolderB/etc/example.file'
POST: https://sharepoint.acme.com:443/sharepoint-site/_api/web/GetFileByServerRelativeUrl(@f)/ContinueUpload(uploadId=guid'680f93eb-8c6a-443d-87ad-1e1b324ea678',fileOffset=3145728)?@f='/sharepoint-site/folderA/subFolderB/etc/example.file'
POST: https://sharepoint.acme.com:443/sharepoint-site/_api/web/GetFileByServerRelativeUrl(@f)/ContinueUpload(uploadId=guid'680f93eb-8c6a-443d-87ad-1e1b324ea678',fileOffset=4194304)?@f='/sharepoint-site/folderA/subFolderB/etc/example.file'
POST: https://sharepoint.acme.com:443/sharepoint-site/_api/web/GetFileByServerRelativeUrl(@f)/ContinueUpload(uploadId=guid'680f93eb-8c6a-443d-87ad-1e1b324ea678',fileOffset=5242880)?@f='/sharepoint-site/folderA/subFolderB/etc/example.file'
POST: https://sharepoint.acme.com:443/sharepoint-site/_api/web/GetFileByServerRelativeUrl(@f)/FinishUpload(uploadId=guid'680f93eb-8c6a-443d-87ad-1e1b324ea678',fileOffset=6291456)?@f='/sharepoint-site/folderA/subFolderB/etc/example.file'

Driven by working on the raw bytes:

final UUID id = UUID.randomUUID();
int offset = 0;
while (offset < bytesTotal) {
    final URI uri;
    if (offset == 0) {
        uri = createStartUri(id, path, filename);
    } else if (offset < bytesTotal - CHUNK_SIZE) {
        uri = createContinueUri(id, offset, path, filename);
    } else {
        uri = createFinishUri(id, offset, path, filename);
    }
    final byte[] bytes = ArrayUtils.subarray(fileDataBytes, offset, offset + CHUNK_SIZE);

    webClient.post().uri(uri).contentType(MediaType.APPLICATION_OCTET_STREAM).contentLength(bytes.length)
            .bodyValue(bytes).exchange().block(Duration.ofSeconds(30));
    
    offset += CHUNK_SIZE;
}

hi @user1016765, Why didnt you go for the approach mentioned by jkemming.Any specific reason? — Amol Kshirsagar, May 08 '22 at 15:56
Because the response came more than a year after I asked the question, and some months after I posted a summary of what I did. I did upvote the response though. — user1016765, May 08 '22 at 22:05

Stream upload 'POST' in Spring WebClient

2 Answers2

Linked