for POST body length of 100000 bytes or above, also with blob/FormData

Question

EDIT: eis was right, there was an interfering (Netbeans) debugger.

The client sends some data to a Spring Boot servlet:

        fetch("/command", {
            method: 'POST',
            cache: 'no-cache',
            headers: {
              'Content-Type': 'application/json'
            },
            body: JSON.stringify(...)
        });

On the server side, HttpServletRequest.getReader() returns a string <Unreadable> (this exact spelling, and without any other error or exception) instead of the JSON string sent by the client. Exactly the same happens for 'Content-Type': 'text/plain; charset=UTF-8'. The JSON string has about 150kB and at a glance, logged to the browser's console before being sent, looks like a single-line JSON string.

For tests, I sent the contents of a few-kB text field instead of the JSON string. This time, the server received an identical string without any problems. Only the JSON string could be sent for some reason. I tested if limiting its size helps. As it turned out, this worked:

body: JSON.stringify(...).substring(0,99999)

but this didn't:

body: JSON.stringify(...).substring(0,100000)

The following:

body: new Blob( [ contentsCallback().substring(0, N) ], { type: 'application/json' } )

works for N<100000. This:

const formData = new FormData();
formData.append('blob', new Blob( [ JSON.stringify(...).substring(0, N) ], { type: 'application/json' } ), 'test');
...
body: formData

works for N<99823.

It looks like there is a size limit somewhere. I do not know if it is on the client or the server side. This is a fragment of server's application.properties where I put various properties discussed here:

multipart.max-file-size=20MB
multipart.max-request-size=20MB
spring.servlet.multipart.max-file-size=20MB
spring.servlet.multipart.max-request-size=20MB
multipart.maxFileSize=20Mb
multipart.maxRequestSize=20Mb
server.tomcat.max-http-post-size=10000000
server.tomcat.max-http-post-size=100000000
server.tomcat.max-swallow-size=100000000

This is how the servlet is registered:

@Bean
public ServletRegistrationBean commandRegistration() {
    ServletRegistrationBean bean = new ServletRegistrationBean(
            new CommandServlet(), "/command");
    // final int SIZE_LIMIT = 20000000;
    // MultipartConfigElement mc = new MultipartConfigElement(TMP_FOLDER, 
    //         SIZE_LIMIT, SIZE_LIMIT * 2, SIZE_LIMIT / 2);
    // bean.setMultipartConfig(mc);
    return bean;
}

Uncommenting the commented part does not change the limit in question.

Here is Chrome data of fetch() which sends the full 150kB JSON (received as <Unreadable>):

Request URL: https://localhost:8443/?command=save
Request Method: POST
Status Code: 200 
Remote Address: [::1]:8443
Referrer Policy: strict-origin-when-cross-origin
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: keep-alive
Content-Length: 0
Content-Type: text/html;charset=UTF-8
Date: Thu, 18 Mar 2021 09:32:00 GMT
Expires: 0
Keep-Alive: timeout=60
Pragma: no-cache
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,pl;q=0.8,de;q=0.7,fr;q=0.6,ru;q=0.5,es;q=0.4,nl;q=0.3,cs;q=0.2,it;q=0.1,nb;q=0.1
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 146111
Content-Type: application/json
Cookie: JSESSIONID=961B41156DCF6F7C57455AA871525198; JSESSIONID=72CCDD7A91C48660B9DD01041F4BEFBB
Host: localhost:8443
Origin: https://localhost:8443
Pragma: no-cache
Referer: https://localhost:8443/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36
command: save

There is neither error nor warning in the console log. The limit is present also when Firefox is used or if I switch the HTTP server from Tomcat to Jetty.

Finally, I searched all sources and all jars (including the dependencies in the local Maven repo) for <Unreadable>. The string cannot be found there.

Can this limit be increased? How to send a large string to a SpringBoot application using the POST method? These answers suggest that practical limits for POST are much larger than 100kB.

based on output provided, you're operating on localhost, right? what kind of operating system? is it possible you have some rogue antivirus, vpn or firewall system that you need to bypass? — eis, Mar 18 '21 at 11:28
only tool I found that would output `` was gdb debugger... so you don't have any debugging tools interfering with your software? — eis, Mar 18 '21 at 11:33
The apps listens at `https://localhost:8443`, the browser (either Chrome or FIrefox) is on the same local system. The system is Linux Mint, derivative of Ubuntu 18.04. I never installed an antivirus or a vpn here and did not even reconfigure iptables, but who knows. — scriptfoo, Mar 18 '21 at 11:34
Are there any diagnostic tools which show how apps in a local system communicate? — scriptfoo, Mar 18 '21 at 11:38
@scriptfoo do you have a possibility to run this app on some other computer than what you are currently using — eis, Mar 18 '21 at 11:38
also, if you hit /command endpoint with similar request but using curl or similar tool, what happens then (given the same input) — eis, Mar 18 '21 at 11:39
@scriptfoo easiest tool might be just running netcat, so shut down your java server, do `nc -l portnumber` with port number that you used your java server with, hit the request and observe what happens — eis, Mar 18 '21 at 11:44
also you could add to the question how CommandServlet actually receives its content, so beginning of method handling input — eis, Mar 18 '21 at 11:47
I have found the kind-of MITM. It was the Netbeans' debugger, incapable of showing strings larger than the limit in question. Thanks a lot for help, it was a hint that the cause had probably nothing to do with web programming. It is funny that @eis has found `` in what was another debugger. Possibly some (cultural) dependency between these two applications. — scriptfoo, Mar 18 '21 at 11:49
@BalusC No, it is a new servlet. I tested if it receives good data and well... it did. I didn't. Yes it was a debug-driven development of `fetch` because my knowledge of Javascript is limited. — scriptfoo, Mar 18 '21 at 12:05
@scriptfoo great! Pls add that as an answer and mark it as accepted so this question would show up as resolved — eis, Mar 18 '21 at 12:09
@scriptfoo at least netbeans/C++ actually uses gdb debugger internally, so it could be the same tool we're talking about, or a clone/port/inspired by the same — eis, Mar 18 '21 at 12:10

score 2 · Answer 1 · answered Mar 18 '21 at 12:29

The commenters suspected that something interferes. One of the hints was a debugger because gdb uses a similar message, only not starting with a capital letter. In fact, the message (disguising as a real variable value) was produced somewhere within the Netbeans/JPDA/JVM debugging chain.

for POST body length of 100000 bytes or above, also with blob/FormData

1 Answers1