3

I am debugging a very odd problem I noticed when testing in my own network. (It doesn't appear to happen on other servers outside my network) where random http requests fail.

I have tried it on two apache servers on my network, and it happens about 1 out of every 100 requests with 2 different apache servers.

I thought it was front-end related, but it appears something to do with my internal network OR configuration. I installed Charles proxy on my machine and used my phone to make the requests in the appellation. (Ajax)

The ajax/http request is being made but it is never making it into my access logs and I am getting the error "Remote Server closed the connection before sending the response header"

http proxy

How can I debug this further?

NOTE: Also worth noting I can only reproduce problem on mobile iPhone + iPad devices when connecting to the server on the machine. (EVEN when using http proxy...which is very odd)

EDIT:

I did a wireshark capture for port 80 from the server computer and accessing it from iPhone. I am having a hard time interpreting it.

Here is the link to the capture files:

CAPTURE 1 (iOS 8 iPhone):

https://www.dropbox.com/s/1ipruv3wlmgng5o/http%20capture%20bad.pcapng?dl=0

NOTE: The error happens after the LAST post to sales/add_payment

CAPTURE 2 (iOS 8 iPhone):

https://www.dropbox.com/s/4zu3654uh9l6230/http%20capture%20bad%202.pcapng?dl=0

NOTE: The error happens after the LAST post to sales/complete

CAPTURE 3 (Android 4.4):

https://www.dropbox.com/s/8xtwkbewce02psw/http%20capture%20android%201.pcapng?dl=0

FOLLOW UP:

If it is indeed faulty network equipment, how do I determine what is bad? (device, router, modem?)

Chris Muench
  • 17,444
  • 70
  • 209
  • 362
  • 2 options usually: either it's a firewall(-like) process that blocks that request, or something makes a webserver(thread) crash... – Wrikken Sep 10 '14 at 20:19
  • How can I check if the web server crashed? It is just me on the server. The very odd part is that it happens very rarely when using the mac as the http proxy. I haven't been able to get it to happen JUST from the mac. I can get it to happen a lot from iOS 8 devices. The error message would seem to indicate it is a server error; but to me it seems like a client having a problem. NOTE: I am using iOS 8 GM. – Chris Muench Sep 10 '14 at 20:23
  • Well: if there's no firewall in between, we can rule out that. Webserver threads don't crash without leaving logs usually, so if there's nothing in the error logs there, we can assume for the moment it's not likely to be that. If you want to go into the nitty-gritty of it, I would run a `tcpdump` / wireshark kind of thing on the webserver to see what it receives & responds on a TCP level. – Wrikken Sep 10 '14 at 20:28
  • I will give that a try. Could it I any way be related to my router or computer? If I connect do a differnet server on differnet machie I don't have the problem. But on two differnet apaches on the same machine it happens. I am upgrading my os to Mac os x 10.10 and will try again. – Chris Muench Sep 10 '14 at 21:09
  • @Wrikken I did wireshark and posted the result in the question. How do I interpret it? – Chris Muench Sep 11 '14 at 13:44
  • [TCP Dup acks tend to indicate broken network equipment](http://serverfault.com/questions/266764/what-causes-duplicate-ack-records) – Wrikken Sep 11 '14 at 13:59
  • @Wrikken If I use an Android device on the network I never have the problem. If I use iOS 8 devices (especially iPhone 4s, I always have the problem). Are their any other possibilities? It also seems to not happen if I am connecting outside network – Chris Muench Sep 11 '14 at 14:02
  • Your Android device might simply be more robust for network failures, this doesn't mean the network isn't failing. Can you get a tcpdump for that Android devices communication and check it for those dup acks? – Wrikken Sep 11 '14 at 14:05
  • I see a couple TCP dup Acks but it doesn't completely fail like iOS. See question for dump – Chris Muench Sep 11 '14 at 14:22
  • @Wrikken I did a reset network settings for my iPhone and it hasn't happened since. I also restarted my router/time capsule. This is a very odd error....but I hope this fixes it. – Chris Muench Sep 11 '14 at 20:34
  • Spoke too soon. Happened again...same type of packet issues. – Chris Muench Sep 11 '14 at 20:38
  • Well, it costs... but replacing hardware might be the quickest way to resolve the issue. However, low cost initially would also be possible: first identify all hardware in use when having problems, and identify hardware in use without having any problems. Note that on routers and switches, specific ports should be regarded seperately here. From low cost to higher: (1) switch your apache servers to a known good port on routers / switches , if they have multiple nics, try to start using another nic on the server (2) borrow iPhone & iPad devices from others to see if they have the same issue. – Wrikken Sep 11 '14 at 21:13
  • (3) start replacing cables (4) start replacing switches (5) replace routers (6) replace servers. (Note that replacing not necessarily means buying, known good hardware from other locations or on loan can be used). – Wrikken Sep 11 '14 at 21:14
  • @Wrikken based on your experience do you see this in any way being caused by the application level? The application never seems to get a valid response (when the error happens). – Chris Muench Sep 12 '14 at 01:15
  • Never on application level. I did see some problems in core OS ip-stacks though, but those are rare. – Wrikken Sep 12 '14 at 13:28
  • see [here](https://stackoverflow.com/questions/831447/the-remote-host-closed-the-connection-in-response-outputstream-write/71270909#71270909) – mfaani Feb 25 '22 at 19:35

0 Answers0