1

I used editcap with option "-d" on a large pcapng file to delete duplicate packets (source file 11GB, new file 5 GB). After that, i extracted all contained files from both pcapng-files (with Networkminer free). I assumed there would be no data loss. Instead, as I compared the two extraction folders (AssembledFiles), I found out that about 30 files were missing in one of them. How is this to explain? Why should deleting duplicate packets result in data loss?

I downloaded Wireshark from "https://www.wireshark.org/download.html" File: Windows Installer (64-bit) - Stable Release 2.6.6 - (v2.6.6-0-gdf942cd8)

Then I got editcap (v 2.6.6.0) from the Wireshark installation folder. <<...... c:\Program Files\Wireshark\ ......>>

Batch:

<<...…

@echo off
echo.
echo.
echo 1. print editcap version
echo.
"c:\Program Files\Wireshark\editcap.exe" -V
echo.

echo.
echo 2. removing duplicate packets from File.pcapng
echo.
"c:\Program Files\Wireshark\editcap.exe" -d "File.pcapng" "File_nodup.pcapng"
echo.

echo.
echo 3. converting File.pcapng and File_nodup.pcapng to pcap format
echo (to make it readable for NetworkMiner free, forgot to mention this step...)
echo.
"c:\Program Files\Wireshark\editcap.exe" -F pcap "File.pcapng" "File.pcap"
echo.
"c:\Program Files\Wireshark\editcap.exe" -F pcap "File_nodup.pcapng" "File_nodup.pcap"
echo.

echo done.

pause
......>>

Result:
<<...... Editcap (Wireshark) 2.6.6 (v2.6.6-0-gdf942cd8)

Copyright 1998-2019 Gerald Combs and contributors. License GPLv2+: GNU GPL version 2 or later http://www.gnu.org/licenses/old-licenses/gpl-2.0.html This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled (64-bit) with GLib 2.42.0, with zlib 1.2.11.

Running on 64-bit Windows 10, build 17763, with Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz (with SSE4.2), with 7841 MB of physical memory, with locale C, binary plugins supported (1 loaded).

Built using Microsoft Visual Studio 2017 (VC++ 14.12, build 25835).

  1. removing duplicate packets from File.pcapng

13625734 packets seen, 6814005 packets skipped with duplicate window of 5 packets.

  1. converting File.pcapng and File_nodup.pcapng to pcap format
    (to make it readable for NetworkMiner free, forgot to mention this step...)

done.
......>>

New Batch:

@echo off
echo.
echo.
echo 1. converting File.pcapng to pcap format
"c:\Program Files\Wireshark\editcap.exe" -F pcap "File.pcapng" "File.pcap"
echo.

echo.
echo 2. removing duplicate packets from File.pcap (parameter "-F pcap" is needed to avoid output to be reconverted to pcapng...)
"c:\Program Files\Wireshark\editcap.exe" -d -F pcap "File.pcap" "File_nodup.pcap"
echo.

echo done.

Tex
  • 29
  • 6
  • Sounds like a great question to ask on the WIreshark support forum. – David Hoelzer Feb 21 '19 at 15:12
  • What version of editcap are you running (i.e., "editcap -V" output)? Perhaps there's a bug and updating Wireshark and its suite of tools to a newer version might help. If you can reproduce this problem with a subset of packets, you can try opening a Wireshark bug report at https://bugs.wireshark.org/bugzilla/ and attaching the smaller file, assuming you can share it, to see if a bug can be confirmed and if so a fix provided by someone. – Christopher Maynard Feb 21 '19 at 18:17
  • @David Hoelzer I am new to packet sniffing and I have just learned why there are (possibly) duplicated packets in a capture file. I honestly did not expect it could be a bug. I just wanted to assure, there are no known scenarios where data loss can occur on deleting duplicates. – Tex Feb 21 '19 at 22:18
  • Can you try using `editcap` to first convert the **.pcapng** file to **.pcap** using something like "`editcap -F pcap File.pcapng File.pcap`", and then de-dup the **.pcap** file to see if you get the exact same results as with the **.pcapng** file or not? Perhaps there's some problem in editcap with de-duping **.pcapng** files, but not with **.pcap** files. Please share your results. – Christopher Maynard Feb 21 '19 at 22:57
  • I'm not saying it's a bug. I'm pointing out that there are dedicated support forums from that project and, while I didn't state it before, this clearly isn't a programming question. – David Hoelzer Feb 22 '19 at 03:28
  • @Christopher Maynard I run a new test converting first "File.pcapng" to "File.pcap", and after that creating File_nodup.pcap, with: "...editcap.exe" -d -F pcap "File.pcap" "File_nodup.pcap" ("-F pcap" is needed to avoid output to be reconverted to pcapng). All created files and all afterwards extracted files were identical at bit level, to those from the first test. I still have no explanation (neither logical nor technical) why the data extraction from the two files should not be identical. – Tex Feb 22 '19 at 05:40
  • @David Hoelzer It's not a question about programming, but a question for programmers. – Tex Feb 22 '19 at 05:54
  • I think your best course of action is to file a Wireshark bug report and attach a capture file that can be used for testing. You'll need to reduce the capture file size though, as the Wireshark bugzilla tracker imposes a 10MB file size limit. If you can't manage to pair down the capture file so it's under that limit, then you'll need to make it available at some other online file sharing site and post a link to it in the bug tracker. Lastly, if further discussion about this topic is needed, you can head over to https://ask.wireshark.org/questions/; I and others are active there. – Christopher Maynard Feb 22 '19 at 14:22
  • @Christopher Maynard I'll see what I can do because I'm not yet authorized to share the data with others. But thank you anyway for your support, your information and your advice. – Tex Feb 25 '19 at 23:28

0 Answers0