3

This problem seems to have started a few days ago without any code changes, my local tomcat server would just stop responding.

My main filter breakpoints don't get hit, and only some resources get loaded.

The ones that do fail, seems to have duplicate response headers:

Accept-Ranges:bytes
Accept-Ranges:bytes
Content-Disposition:inline;filename="online-framework.js"
Content-Disposition:inline;filename="online-framework.js"
Content-Encoding:gzip
Content-Encoding:gzip
Content-Range:bytes 0-42062/42063
Content-Range:bytes 0-42062/42063
Content-Type:application/javascript
Content-Type:application/javascript
Date:Tue, 19 Apr 2016 13:01:18 GMT
ETag:online-framework.js_42063_1461060446000
ETag:online-framework.js_42063_1461060446000
Expires:Tue, 26 Apr 2016 13:01:19 GMT
Expires:Tue, 26 Apr 2016 13:01:19 GMT
Last-Modified:Tue, 19 Apr 2016 10:07:26 GMT
Last-Modified:Tue, 19 Apr 2016 10:07:26 GMT
Server:Apache-Coyote/1.1
Server:Apache-Coyote/1.1
Transfer-Encoding:chunked
Transfer-Encoding:chunked
Vary:Accept-Encoding
Vary:Accept-Encoding

And chrome stops with an error: Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING, which makes sense.

The tomcat connector has not been modified and does include a gzip response. what have I tried:

  • thread dump - no obvious blocking code
  • wireshark - don't know it good enough to figure out what goes wrong
  • disabled antivirus
  • no proxies running

The page just seems to load forever. Here is the kicker, it happens randomly, it would start working one hour but not the next.

The only thing I can thinl of is that some domain thing may have changed, no windows updates have run, and 3 or 4 other pc's do the same thing while 2 others do not (same build)

I have no idea where to look next? Any ideas?

epoch
  • 16,396
  • 4
  • 43
  • 71

1 Answers1

2

Ok guys, answering my own question here, but I have found the solution.

I was using a custom file servlet written by @BalusC; and the problem was therein

Here are my findings:

  • The issue appears when using a combination of Content-Encoding: gzip and Content-Range
  • The resulting error is: ERR_INCOMPLETE_CHUNKED_ENCODING
  • I first decided to disable this filter and let tomcat's DefaultServlet handle it... problem gone
  • being a programmer, I had to know why.
  • I still don't have the exact reason, but I think it is because gzip cannot be accurately represented with a length

The spec for Content-Range also states that:

The Content-Range entity-header is sent with a partial entity-body to specify where in the full entity-body the partial body should be applied. Range units are defined in section 3.12.

And within the code, it was sent even if it was the full response:

if (ranges.isEmpty() || ranges.get(0) == full) {
    // Return full file.
    Range r = full;
    response.setContentType(contentType);
    response.setHeader("Content-Range", "bytes " + r.start + "-" + r.end + "/" + r.total);

    if (content) {
        // .....

I removed that line, and everything started working again! I would really like someone to chip in on this and possibly give a better explanation.

Here is the chrome://net-internals/ output of a file that failed:

t= 3740 [st=   38]     -HTTP_STREAM_REQUEST
t= 3740 [st=   38]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]
t= 3740 [st=   38]        HTTP_TRANSACTION_SEND_REQUEST_HEADERS
                          --> GET /Core/resources/scripts/scriptaculous/dragdrop.js?t=1461139610 HTTP/1.1
                              Host: localhost:8080
                              Connection: keep-alive
                              Pragma: no-cache
                              Cache-Control: no-cache
                              Accept: */*
                              User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
                              DNT: 1
                              Referer: http://localhost:8080/Core/Dashboard?componentID=VCmq3c
                              Accept-Encoding: gzip, deflate, sdch
                              Accept-Language: en-US,en;q=0.8,af;q=0.6
                              Cookie: [306 bytes were stripped]
t= 3740 [st=   38]     -HTTP_TRANSACTION_SEND_REQUEST
t= 3740 [st=   38]     +HTTP_TRANSACTION_READ_HEADERS  [dt=4]
t= 3740 [st=   38]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=4]
t= 3744 [st=   42]        HTTP_TRANSACTION_READ_RESPONSE_HEADERS
                          --> HTTP/1.1 200 OK
                              Server: Apache-Coyote/1.1
                              Content-Disposition: inline;filename="dragdrop.js"
                              Accept-Ranges: bytes
                              ETag: dragdrop.js_19250_1461136271305
                              Last-Modified: Wed, 20 Apr 2016 07:11:11 GMT
                              Expires: Wed, 27 Apr 2016 08:06:51 GMT
                              Content-Range: bytes 0-19249/19250
                              Content-Type: application/javascript
                              Transfer-Encoding: chunked
                              Vary: Accept-Encoding
                              Date: Wed, 20 Apr 2016 08:06:50 GMT
t= 3744 [st=   42]     -HTTP_TRANSACTION_READ_HEADERS
t= 3744 [st=   42]      HTTP_CACHE_WRITE_INFO  [dt=56]
t= 3800 [st=   98]      HTTP_CACHE_WRITE_DATA  [dt=0]
t= 3800 [st=   98]      HTTP_CACHE_WRITE_INFO  [dt=1]
t= 3801 [st=   99]      URL_REQUEST_DELEGATE  [dt=0]
t= 3801 [st=   99]   -URL_REQUEST_START_JOB
t= 3801 [st=   99]    URL_REQUEST_DELEGATE  [dt=0]
t= 3801 [st=   99]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t= 3801 [st=   99]    HTTP_CACHE_WRITE_DATA  [dt=1]
t= 3802 [st=  100]    URL_REQUEST_JOB_BYTES_READ
                      --> byte_count = 3683
t= 3802 [st=  100]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t= 3802 [st=  100]    HTTP_CACHE_WRITE_DATA  [dt=0]
t= 3802 [st=  100]    URL_REQUEST_JOB_BYTES_READ
                      --> byte_count = 13982
t= 3802 [st=  100]    HTTP_TRANSACTION_READ_BODY  [dt=20365]
                      --> net_error = -355 (ERR_INCOMPLETE_CHUNKED_ENCODING)
t=24167 [st=20465]    FAILED
                      --> net_error = -355 (ERR_INCOMPLETE_CHUNKED_ENCODING)
t=24168 [st=20466] -REQUEST_ALIVE
                    --> net_error = -355 (ERR_INCOMPLETE_CHUNKED_ENCODING)

And, finally, here are some links that really helped me, seems spring had the same problem last year.

I still cannot figure out why this randomly started after running fine for years and would really appreciate any input.

Community
  • 1
  • 1
epoch
  • 16,396
  • 4
  • 43
  • 71
  • Nice find. I suspect a Chrome bug. Does this also occur with other browsers or older Chrome versions? Nonetheless, `Content-Range` indeed doesn't belong there when it represents the full range, but it should have been harmless as it's the `Content-Length` which is leading for the content length. By the way, the old FileServlet has been reworked to http://showcase.omnifaces.org/servlets/FileServlet from which Gzip response was stripped off and split to http://showcase.omnifaces.org/filters/GzipResponseFilter Do they work well in your case? – BalusC Apr 20 '16 at 12:45
  • @BalusC, thank you very much for your input! It happened in Firefox & Chrome, which confused me. I already saw the new reworked `FileServlet`, looks awesome, i'm going to make some time to implement them and see if the problem still occurs when using them in conjunction with each other :) – epoch Apr 20 '16 at 13:08
  • Okay, just to be on safe side I removed the unnecessary `Content-Range` on full file from both the blog and OmniFaces. The `GzipResponseFilter` by the way already skips responses with `Content-Range` header from gzipping. The old `FileServlet` should after all have done the same. – BalusC Apr 20 '16 at 13:23
  • Great news, I actually just wanted an authoritative comment; I saw in Spring, they basically just disabled ranges when using gzip (if I understood the tickets correctly) – epoch Apr 20 '16 at 13:43
  • 1
    Theoretically, it should not have been necessary. Therefore I suspect something went wrong in client (or proxy) side. – BalusC Apr 20 '16 at 13:49
  • most likely, I still can't exactly finger it, but since it started happening sporadically this week and has been running fine for years, I suspect a software upgrade / network change of some kind. – epoch Apr 20 '16 at 13:53
  • 1
    FYI: this workaround has been further improved as per https://github.com/omnifaces/omnifaces/issues/269 – BalusC Jun 20 '16 at 18:58