Spring Boot multipart/form-data request file streaming to downstream service

Question

I have a microservice architecture, where one service acts as a proxy, and must only forward the uploaded form data payload to the downstream service using restTemplate, preferably without loading anything from the request on disk or into memory.

score 1 · Answer 1 · answered Jan 26 '22 at 09:16

I managed to resolve the issue taking the following steps. Here I will describe the approaches, and the limitations used:

I have the following rest template configuration:

@Bean
public RestTemplate myRestTemplate() {
    HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
    requestFactory.setBufferRequestBody(false);
    RestTemplate restTemplate = new RestTemplate(requestFactory);
    restTemplate.setInterceptors(new ArrayList<>()); // to avoid interceptors loading data into memory
    return restTemplate;
}

in my controller I am processing the HttpServletRequest directly using Apache Commons FileUpload Streaming Api with one asterix: Special care on the multipart form data, so first the form fields are processed in the while loop, and then only one file was I able to process, since:

 FileItemStream fileItemStream = uploadItemIterator.next();
 return fileItemStream.openStream();

must be returned without invoking itemIterator.hasNext(), because that will result in FileItemStream.ItemSkippedException which works wonderfully, no data is saved on disk

c:\Users\myuser\AppData\Local\Temp\tomcat.11416588345568217859.8077\

note: I have set the following property as stated in the documentation.

spring.application.servlet.multipart.enabled: false

From here, Using the streaming api I have an inputStream, which I will pass further down to create my HttpEntity as follows (simplified in example, full inspiration to include filename in request: here):

    MultiValueMap<String, Object> multiPartBody = new LinkedMultiValueMap<>();
    multiPartBody.add(FILE, inputStream);
    HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(multiPartBody, myHeaders);

after this, I do make the call to my rest Template:

    myRestTemplate.postForEntity(url, requestEntity, MyResponse.class);

this goes all the way via the following sequence:

RestTemplate.doExecute()
  HttpAccessor.createRequest()
    HttpComponentsClientHttpRequestFactory.createRequest() -> which will return a **HttpComponentsStreamingClientHttpRequest** <- this one is important
      RestTemplate.doWithRequest(ClientHttpRequest httpRequest) -> calls: ((HttpMessageConverter<Object>) messageConverter).write(
                                requestBody, requestContentType, httpRequest);
        FormHttpMessageConverter.write()
          FormHttpMessageConverter.writeMultipart() -> where outputMessage instanceof StreamingHttpOutputMessage is true
            HttpComponentsStreamingClientHttpRequest.executeInternal -> creates a new StreamingHttpEntity(...)
              after which this goes down on InternalCLientExecution, and in execChain

sooner or later it will enter in the chain:

HttpComponentsStreamingClientHttpRequest.StreamingHttpEntity.writeTo(OutputStream outputStream) throws IOException {
    this.body.writeTo(outputStream);
}

where body is a FormHttpMessageConverter.lambda from above:

if (outputMessage instanceof StreamingHttpOutputMessage streamingOutputMessage) {
    streamingOutputMessage.setBody(outputStream -> {
        writeParts(outputStream, parts, boundary);
        writeEnd(outputStream, boundary);
    });
}

so we get further down, and end up in:

FormHttpMessageConverter.writeParts()
    FormHttpMessageConverter.writePart()

here a multipartMessage is composed and passed further down (or invoked the superclass AbstractHttpMessageConverter method)

multipartMessage = new MultipartHttpOutputMessage(os, charset);
...
((HttpMessageConverter<Object>) messageConverter).write(partBody, partContentType, multipartMessage);

from here we get into AbstractHttpMessageConverter.write where condition

if (outputMessage instanceof StreamingHttpOutputMessage)

evaluates to false because MultipartHttpOutputMessage is not an instance of StreamingHttpOutputMessage

But this seems not to affect anything, since the whole thing is invoked in the above mentioned lambda, sooner or later, we need to write the bytes from the inputStream into the outputStream.

one impediment:

if I configure the restTemplate as follows:

@Bean
@org.springframework.cloud.client.loadbalancer.LoadBalanced
public RestTemplate myRestTemplate() {
...
}

there is an interceptor/aspect overriding the RestTemplate HttpComponentsClientHttpRequestFactory with RibbonClientHttpRequestFactory (using spring netflix stack), which does not support setBufferRequestBody(false).

That is how I managed to solve the file streaming issue, hope it helps others too: Limitations/Constraints:

You cannot use MultipartFile in your controllers since spring by default saves data into temp files on fileSystem (can't use resolve-lazily either: because), I was able to overcome this issue only with Apache Commons FileUpload
Using Apache Commons FileUpload I managed to process only one file, and the form data need to be processed before the file data
spring.application.servlet.multipart.enabled: false -> affects other endpoints too
composing downstream form data with correct Content-Disposition: form-data; name="file"; filename="my.txt" needs some strange embedded HttpEntity constructions
@LoadBalanced overrides the whole restTemplate requestFactory

Good luck everyone, and any feedback is welcome.

Spring Boot multipart/form-data request file streaming to downstream service

1 Answers1