There is a number of anwsers for the question on how to make RestTemplate stream the request.
Why does RestTemplate consume excessive amounts of memory?
POST InputStream with RestTemplate
It looks like simple by setting the buffering to false on the requestfactory. Well, now after 3 days of debugging I think it just never actually supports streaming at all. If anyone can explain me why following explanation is wrong, it would be very helpful.
I am using springframework 5.3.19. Maybe it only does not work in that version? And true, I should use WebClient but at first I wanted to understand why I am the only one on the planet not having a working solution...
So what happens when using spring RestTemplate. Here is some pseudocode to explain what is happening on a high level:
Request request = create(args); // creates the intercepting request
messageConverters.prepare(request); // uses the intercepting request!!
request.execute(); // create the actual request as underlying delegate
By definition RestTemplate is a InterceptingHttpAccessor, which means it supports interceptors by having its own wrapping requestFactory (InterceptingClientHttpRequestFactory). Imho not the best practice, but ok this can be debatable. At first there is nothing problematic here, if it did not create its own request type (InterceptingClientHttpRequest). That request type is by definition a AbstractBufferingClientHttpRequest. Well, we try to stream here...
Ok, no worries, it behaves a bit like a proxy to the actual request... Well, not really. Only when the request is "executed" it will instantiate the actual underlying (streaming) request type. Ironically the InterceptingClientHttpRequest is used for preparing the request body when writing it via a message converter (which happens before executing the request). In its superclass AbstractHttpMessageConverter it checks on whether the request is instanceof StreamingHttpOutputMessage, and in that case it prepares for streaming.
But that code will never be executed since on this level the request is always an InterceptingClientHttpRequest, not a StreamingHttpOutputMessage.
I tried following to bypass this behavior: I made sure the body was set on a HttpComponentsStreamingClientHttpRequest but that was simply not working because underlying the outputstream to the request was a ContentLengthOutputStream, which was actually just ignoring all the written bytes because it checks whether the length of the array of bytes is less than the content length (which is set to some default for streaming, genre 0 or -1). Anyhow the class StreamingHttpEntity inside HttpComponentsStreamingClientHttpRequest returns false for property chunked (which is something that I would expect as the value to flush chunk by chunk).
And as far as I can see the SimpleStreamingClientHttpRequest via the SimpleClientHttpRequestFactory is unlikely to work since it has no link with the resource. It could maybe work if it was already created when interfacing with the message converter, but that is not the case, see earlier explanation. Also, a SimpleStreamingClientHttpRequest is not a StreamingHttpOutputMessage while HttpComponentsStreamingClientHttpRequest is in fact.
Just a last note: the way I found out it is not streaming at all is because I had to send a file of more than 2.2 GB, which is basically the limit of a byte array in Java because its length can only be MAX_INT long. And the ByteArrayOutputStream used has that as max size (obviously), with a check boiling down to ArraysSupport.hugeLength. Which throws imho wrongly a OutOfMemoryError when you exceed that length. That should have been a RuntimeException. At first I was confused, because I actually thought the JVM had a heap size issue (which was completely not the case).
So anyone any idea where I'm doing the wrong thing. Wrong assumptions? Thanks in advance.
