1

enter image description here

I'm working on a http communication project. The goal is to exchange http data (for exemple JSON data) between a PIC18F67j60 microcontroller (the PIC18 embarks on a Ethernet module) and a host computer via an Ethernet link. The computer acts as the client and the PIC18 as the server. I'm using MPLABX IDE programming with C language. I also use Wireshark to have a precise look on the frames exchanged between the PIC and the computer. I already managed to set up a TCP connection between both PIC18 and host computer (I can see it on Wireshark, that displays the 3-way handshake process correctly). To make it simple, here is how I test the http communication : I run firefox, and type on it : "@IP of my microcontroller : Port number". So this has firefox to send a http GET request ("GET HTTP/1.1 .." )and then my microcontroller receives and replies by sending "HTTP/1.1 200 OK ..." to Firefox. And I can definitely see this reply ("HTTP/1.1 200 OK ..") on wireshark which indicates that the replying frame sent from the PIC18 is http type (see picture below). But then it seems like Firefox doesnt want to display the data content (JSON data) and instead sends a TCP Keep Alive request that I dont understand at all. Does someone know why Firefox doesnt want to display the JSON Data and instead keeps on sending TCP Keep Alive request ? Or Am I missing something in http exchanging ? . PS : for those who maybe dont know, wireshark can analyze frames and indicate the issue that is going on with these frames, so a colored line (see picture below) is associated to a certain type of issue. More precisely, on the picture, from what Wireshark says, the purple lines are associated with "BAD TCP : TCP.analysis.flags" issue. Thank you for all your help.

enter image description here

Steffen Ullrich
  • 114,247
  • 10
  • 131
  • 172
Idash
  • 55
  • 8
  • TCP keep alive is irrelevant for this. And how exactly do you expect Firefox to display JSON? It can display HTML and text and images directly, but what exactly should it do with JSON? JSON is typically used inside some web application which then extracts the information out of it and renders it as HTML. – Steffen Ullrich Aug 28 '20 at 07:44
  • Thank you @SteffenUllrich for your response. I just want Firefox to display the data in its raw form like this : { "key" : "value"} – Idash Aug 28 '20 at 12:04
  • so in the text form – Idash Aug 28 '20 at 12:17

1 Answers1

1

If you want to browser display the data as text you need to use the correct value in the Content-Type header. Currently you are using application/json which has no inherent meaning for the browser, i.e. it does not know how these data should be displayed. Instead you need to use a Content-Type of text/plain.

Apart from that you seem to use some broken HTTP stack. It does not specify a size of the response body (using Content-length) but also does not close the connection after the response is sent. This means it is unknown to the client (Firefox) when the response is done and it will keep waiting - and not display anything.

Steffen Ullrich
  • 114,247
  • 10
  • 131
  • 172
  • Thank you @SteffenUllrich, I tried this but its still not working. Here is what I can see from WireShark, and it seems like everythying is received correctly by Firefox but it still not displaying the data text : "HTTP 200 OK \r\n Connection: close \r\n Content-Type: text/html\r\n Cache-Control: no-cache\r\n \r\n {"key":"ok"} – Idash Aug 28 '20 at 14:05
  • @Idash: Why did you use `text/html` although I clearly said `text/plain` ? Also, this does not look like a correct HTTP response at all (i.e. it should be `HTTP/1.1 200 ..` and not `HTTP 200 ..`). You also don't have any `Content-Length` but you seem to keep the connection open - which makes it impossible to detect when the response ends. Is this some self-written (and broken) HTTP stack on the PIC18? – Steffen Ullrich Aug 28 '20 at 14:44
  • I'm completely sorry, I just made a mistake typing the response so the correct frame Wireshark sees (and that i am sending all the way from the PIC18) is : HTTP/1.1 200 OK\r\n Content-Type: text/plain\r\n Cache-Control: no-cache\r\n \r\n {"key":"ok"} – Idash Aug 28 '20 at 14:57
  • And despite all of this, firefox keeps running in search for the data and keeps sending some TCP request as if he didnt receive anything from the PIC. Finally Firefox ends up displaying "Error" @Steffen Ullrich – Idash Aug 28 '20 at 14:59
  • @Idash: Again: *"You also don't have any Content-Length but you seem to keep the connection open - which makes it impossible to detect when the response ends. Is this some self-written (and broken) HTTP stack on the PIC18?"*. Firefox is waiting for more since your response does not specify where the response ends. See also updated answer. – Steffen Ullrich Aug 28 '20 at 15:12
  • I already tried to add both « Content-Length: 12 » and « Connection: close ». But Firefox doesnt display antything :/ – Idash Aug 28 '20 at 20:59
  • @Idash: Can you please provide a packet capture (pcap, not just image) so that one can check what is really happening? I assume that the problem is in something you currently don't show. – Steffen Ullrich Aug 29 '20 at 06:05
  • what a pcap is please ? I’m not really advanced with Wireshark, I just have some basics. Actually the image that I posted is from a capture I mean, I put a IP filter ( in order to only see the exchanges between my PIC and my computer) and then I run the capture and then pause it and take a screenshot of it. – Idash Aug 29 '20 at 08:04
  • @Idash: pcap is a file with captured packets. See for example [here](https://stackoverflow.com/questions/7992581/saving-the-displayed-filtered-packets-in-wireshark) how to export what you have in Wireshark as pcap. – Steffen Ullrich Aug 29 '20 at 08:31
  • Oh ok thank you a lot, I’ll give you the file the day after tomorow since I dont have the PIC right now to re run the project – Idash Aug 29 '20 at 09:09
  • @Idash: You can upload the pcap to some file sharing service and then provide a link in your question or in a comment here. And please refrain from editing your question or adding answers as a way to get into a dialog with me - that's what comments are for. – Steffen Ullrich Aug 31 '20 at 12:05
  • this is the only way I found to share the pcap file : https://github.com/camour/Wireshark_Network.git Once you clicked on the link, click on "capture.pcapng", then on "view raw" (view raw is displayed at the bottom) of the project – Idash Aug 31 '20 at 12:13
  • @Idash: Thanks. Where did you capture this pcap since it looks really strange. Based on this the clients TCP stack did not receive the servers response (no ACK) and seems to assume that the server acked only up to sequence 355 while the pcap shows that the server acked up to 356. This does not make any real sense. – Steffen Ullrich Aug 31 '20 at 12:31
  • Ok but how did you figure that the client (in our case Firefox) did not receive the server's response (in our case the microcontroller's response) ? To me, the fact the server is acking 356 bytes to the client makes sense since the acq sequence started on 1 (due to 3-way handshake phase where acq is set to "1" even before data exchanging) – Idash Aug 31 '20 at 12:57
  • @Idash: if you look at packet 7..10 you see that the client does not ACK the servers response - which suggests that it was not received. Instead it tries to deliver a single byte seq 356 although this byte was already acked by the server. And it is not even the same byte as in the original request. So something is really strange here in the network. – Steffen Ullrich Aug 31 '20 at 13:21
  • here is another capture that seems less weird in the data Exchange process. You'll find out the client does not receive the HTTP response and thus keeps on sending "the GET HTTP request " through TCP Spurious Retransmission. I suspect the issue coming from a timeout (Client does not receive an ACK response before a timeout so it sends back the HTTP request): https://github.com/camour/Wireshark_Network.git @SteffenUllrich – Idash Aug 31 '20 at 14:19
  • @Idash: Looks like a similar problem. Where exactly do you capture the traffic - at the client side (browser) or at the server side or somewhere in between? – Steffen Ullrich Aug 31 '20 at 16:54
  • I am capturing frames on the client side. @SteffenUllrich Maybe I should mention there is an Ethernet switch between the client and the server. (which I dont think has much impact on the exchanges) – Idash Sep 01 '20 at 06:09
  • @Idash: Have you any software installed on the client which might intercept network traffic, like an Antivirus or other alike products? Try to use a clean setup with only a minimal system. – Steffen Ullrich Sep 01 '20 at 06:57
  • How do we get to do this, please ? (Thank you for all your help I really appreciate). @SteffenUllrich – Idash Sep 01 '20 at 07:44
  • I got something really strange to me, so remember what I am trying to do is : I sent an HTTP request from Firefox to my server. The TCP connection is set correctly, then Firefox sends out a HTTP request to get some data but when it comes to the server's reply, something is going wrong, causing Firefox to keep sending http requests as if Firefox receives nothing – Idash Sep 01 '20 at 09:14
  • Well, at this particular moment, I stop Firefox to prevent him from sending other requests, which causes Firefox to close the TCP connection that he previously established with my server (my PIC). Right after I can see on Wireshark that Firefox is sending the FYN-ACK flags to the server and more precisely I can see that Firefox **"seq number"** is totally correct. That implies Firefox has finally received my Server's ACKNOLEDGMENT (the ACK that acked the HTTP GET Request sent from firefox) – Idash Sep 01 '20 at 09:14
  • I finally suspect the origin of my issue to be related to the **Spurious Retransmission**, as my pic HTTP response delay exceeds my client's timeout, the client doesnt accept the packet and thus retransmits a request. It explains why SYN et FIN packets are accepted (because they dont take much time to be sent from my PIC) by my client but not my http packets – Idash Sep 01 '20 at 14:21
  • @Idash: I'm not convinced. The ACK and reply from the server comes immediately but from the ACK in the following client packets it looks like that the client does not receive it. Something is blocking the client from receiving the servers response, it is not a timing problem. – Steffen Ullrich Sep 01 '20 at 15:08
  • But why the SYN and FIN flag packets would be accepted by the client and not the HTTP packet ? Or you mean maybe the client is not allowed to receive HTTP data from my PIC ? And please what is a clean set up, how can I proceed it ? @SteffenUllrich – Idash Sep 01 '20 at 21:50
  • @Idash: A clean set is a computer with only basic OS and browser installed, no antivirus or whatever software you might have installed. This is to make sure that the problems you see do not come from something installed on your system. You might also use a new system to have a better comparison to the old one. Still, the symptoms here I've never seen before. – Steffen Ullrich Sep 02 '20 at 07:04
  • By any chance, is it because of my HTTP header that I wrote using ASCII code ? – Idash Sep 02 '20 at 08:37
  • Hey there! I finally figured what was the issue. I really apologize. It was because of my TCP Checksum Calculation, I just hadnt understand how TCP Checksum was compute, actually I thought the TCP Checksum was only compute over the TCP Header and the TCP **PSEUDO** Header **without data** ! But I had to take the data into account in the TCP Checksum calculation ! Thank you for all your help, I really apologize for taking your time @SteffenUllrich – Idash Sep 02 '20 at 09:28
  • @Idash: This was unexpected. But good that you've figured it out at the end. – Steffen Ullrich Sep 02 '20 at 16:13