I try to capture an HTTP-download with Python using dpkt and pcap. The code looks like
...
pc = pcap.pcap(iface)
for ts, pkt in pc:
handle_packet(pkt)
def handle_packet(pkt):
eth = dpkt.ethernet.Ethernet(pkt)
# Ignore non-IP and non-TCP packets
if eth.type != dpkt.ethernet.ETH_TYPE_IP:
return
ip = eth.data
if ip.p != dpkt.ip.IP_PROTO_TCP:
return
tcp = ip.data
data = tcp.data
# current connection
c = (ip.src, ip.dst, tcp.sport, tcp.dport)
# Handle only new HTTP-responses and TCP-packets
# of existing connections.
if c in conn:
handle_tcp_packet(c, tcp)
elif data[:4] == 'HTTP':
handle_http_response(c, tcp)
...
In handle_http_response() and handle_tcp_packet() i read the data of the tcp-packets (tcp.data) and write them to a file. However i noticed that i often get packets with the same TCP sequence number (tcp.seq) (on the same connection) but it seems that they contain the same data. Moreover it seems that not all packets are captured. For example if i sum up the packet-sizes the resulting value is lower than the one listed in the http-header (content-length). But in Wireshark i can see all packages.
Does anyone has an idea why i get those duplicate packets and how i can capture every packet belonging to the http-response?
EDIT:
Here you can find the complete code: pastebin.com.
When running it prints something like that to stdout:
Waiting for HTTP-Audio-responses ...
...
New TCP-Packet, len=1440, tcp-payload=5107680, con-len=5197150 , dups=57 , dup-bytes=82080
New TCP-Packet, len=1440, tcp-payload=5109120, con-len=5197150 , dups=57 , dup-bytes=82080
New TCP-Packet, len=1440, tcp-payload=5110560, con-len=5197150 , dups=57 , dup-bytes=82080
----------> FIN <----------
New TCP-Packet, len=1937, tcp-payload=5112497, con-len=5197150 , dups=57 , dup-bytes=82080
New TCP-Packet, len=0, tcp-payload=5112497, con-len=5197150 , dups=57 , dup-bytes=82080
As you can see the TCP-payload plus the duplicate received bytes (5112497+82080=5194577) are lower than the filesize of the download (5197150). Moreover you can see that i receive 57 duplicate packages (same SEQ and same TCP-data) and that still packages are received after the packet with the FIN-flag.
So does anyone have an idea how i can capture all packets belonging to the connection? Wireshark sees all packets and i think it uses libpcap too.
I don't even know if i do something wrong or if the pcap-library does something wrong.
EDIT2:
OK, it seems that my code is correct: In Wireshark I saved the captured packets and used the capture-file in my code (pcap.pcap('/home/path/filename') instead of pcap.pcap('eth0')). My code read perfectly all packages (on multiple tests)! Since Wireshark uses libpcap too (afaik), i think the problem is the lib pypcap which does not provide me all packages.
Any idea on how to test that?
I already compiled pypcap by myself (trunk) but that didn't change anything -.-
EDIT3:
OK, I changed my code to work with pcapy instead of pypcap and have the same problem:
When reading the packets from a previous captured file (created with Wireshark) then everything is fine, but when I capture the packets directly from eth0 I miss some packets.
Interesting: When running both programs (the one using pypcap and the one using pcapy) in parallel they capture different packets. e.g. one programm receives one packet more.
But I have still no idea why -.-
I thought Wireshark uses the same base-lib (libpcap).
Please help :)
