1

I am using kismet running on a raspberry pi to capture network data that I need to analyze in real time on another device. Kismet's api has an endpoint to receive a stream of binary data in the pcapng format. I have successfully been able to read the stream but I cannot find any documentation on getting useful data out of the stream besides the actual pcapng standard. This would be fine but the data coming in is not consistent with the format and I have found no libraries that have been able to successfully parse this stream. I think that the issue is that when reading the stream extra data is inserted between each captured packet so determining the start of the packet is the main issue. I am currently using python but am open to other languages if they can easily solve this problem or have libraries already written.

This is the code I am using to get the data into python where print(line.hex()) would be where I accessed the relevant data.

import requests
r = requests.get(url, stream=True)
for line in r.iter_lines():
    print(line.hex())
b-rad15
  • 89
  • 1
  • 2
  • 13
  • "getting useful data out of the stream" => How does expectation differ from actual? Can you [provide](https://stackoverflow.com/posts/58421323/edit) both in your question? – Ross Jacobs Oct 16 '19 at 20:57
  • Also keep in mind that python's `print` will automatically add a newline char much like bash's `echo` (this may or may not be relevant, depending on what `print(line.hex())` is standing in for). – Ross Jacobs Oct 16 '19 at 21:01
  • Edited to add that I need to find the start of a packet and then can more easily follow the standard and also the print is essentially standing in for `line[bytenumbers]` so no newline affecting anything. – b-rad15 Oct 17 '19 at 00:58
  • Normally, a pcapng file will have a pcapng header (Section Header Block), and packets (Enhanced Packet Block). 'is not consistent with the format' => Please elaborate. – Ross Jacobs Oct 17 '19 at 01:27
  • So i did some more research today and the first issue is that there is no block type on the section header block. After assuming block 1 was the shb I was able to read it fine for the most part but now about half of the blocks have lengths inconsistent with the defined lengths in bytes 4-8. I also did a test and simply did line.split(b'\x06\x00\x00\x00') assuming every block was a packet and going from there the blocks with inconsistent lengths were off by 1 or 6 bytes no more no less. – b-rad15 Oct 17 '19 at 20:40
  • Sounds like a Kismet has a bug with its pcapng implementation - I would [raise an issue](https://github.com/kismetwireless/kismet/issues/new). – Ross Jacobs Oct 17 '19 at 20:44

1 Answers1

2

It looks like the error was that the python requests module was not capturing the entire stream and certain bytes were missing. By using 

curlSubProcess = subprocess.Popen(['curl', 'url/pcap/all_packets.pcapng'], stdout=subprocess.PIPE, bufsize=1)
curlSubProcess.stdout.read(lengthnext)

in a while loop where lengthnext is the length of the next packet I was able to read it correctly. I then parsed this binary block data with a class I wrote and the packet data with scapy and it is working as expected.

b-rad15
  • 89
  • 1
  • 2
  • 13