Here's part of the answer. I can read in your data, chunk-by-chunk. Once you get it in, then you can decide to write it back out into a set of smaller files (using BinaryWriters on FileStreams). I'll leave that to you. But, this reads everything in.
Update: There's more of the answer below (I added the WriteStruct method, and something closer to what you asked for)
I start by defining two structures with very clear layout. Since the header consists of just two consecutive 64 bit uints, I can just use LayoutKind.Sequential:
[StructLayout(LayoutKind.Sequential)]
public struct CanHeader {
public UInt64 TimeStampFrequency;
public UInt64 TimeStamp;
}
But, the Chunk structure mixes and matches 32 and 64 bit uints. If I lay it out sequentially, the framework inserts 4 bytes of padding to align the UInt64s. So, I need to use LayoutKind.Explicit:
[StructLayout(LayoutKind.Explicit)]
public struct CanChunk {
[FieldOffset(0)] public UInt32 ReturnReadValue;
[FieldOffset(4)] public UInt32 CanTime;
[FieldOffset(8)] public UInt32 Can;
[FieldOffset(12)] public UInt32 Ident;
[FieldOffset(16)] public UInt32 DataLength;
[FieldOffset(20)] public UInt64 Data;
[FieldOffset(28)] public UInt32 Res;
[FieldOffset(32)] public UInt64 TimeStamp;
}
Then I took a look at @FelixK's answer to C# array within a struct, and modified his ReadStruct extension method to suit my needs:
private static (T, bool) ReadStruct<T>(this BinaryReader reader) where T : struct {
var len = Marshal.SizeOf(typeof(T));
Byte[] buffer = reader.ReadBytes(len);
if (buffer.Length < len) {
return (default(T), false);
}
//otherwise
GCHandle handle = default(GCHandle);
try {
handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
return ((T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T)), true);
} finally {
if (handle.IsAllocated)
handle.Free();
}
}
It returns a tuple, where the first member is a structure instance that's just been read from the file, and the second member is a flag to indicate whether more reads are needed (true says "keep reading"). It also uses BinaryReader.ReadBytes, rather than BinaryReader.Read.
With all that in place, now I read the data. My first try had me writing things out to the console - but it takes forever to write out 140 MB. But, if you do that, you will see the data moving the way you'd expect (the time stamp keeps going up).
public static void ReadBinary() {
using (var stream = new FileStream("Klassifikation_only_Sensor1_01.dr2", FileMode.Open, FileAccess.Read)) {
using (var reader = new BinaryReader(stream)) {
var headerTuple = reader.ReadStruct<CanHeader>();
Console.WriteLine($"[Header] TimeStampFrequency: {headerTuple.Item1.TimeStampFrequency:x016} TimeStamp: {headerTuple.Item1.TimeStamp:x016}");;
bool stillWorking;
UInt64 totalSize = 0L;
var chunkSize = (UInt64)Marshal.SizeOf(typeof(CanChunk));
do {
var chunkTuple = reader.ReadStruct<CanChunk>();
stillWorking = chunkTuple.Item2;
if (stillWorking) {
var chunk = chunkTuple.Item1;
//Console.WriteLine($"{chunk.ReturnReadValue:x08} {chunk.CanTime:x08} {chunk.Can:x08} {chunk.Ident:x08} {chunk.DataLength:x08} {chunk.Data:x016} {chunk.Res:x04} {chunk.TimeStamp:x016}");
totalSize += chunkSize;
}
} while (stillWorking);
Console.WriteLine($"Total Size: 0x{totalSize:x016}");
}
}
}
If I uncomment the Console.WriteLine statement, the output starts out looking like this:
[Header] TimeStampFrequency: 00000000003408e2 TimeStamp: 000002a1a1bf04bb
00000001 a1bf04bb 00000020 000002ff 00000008 0007316be2c20350 0000 000002a1a1bf04bb
00000001 a1bf04be 00000020 00000400 00000008 020a011abf80138e 0000 000002a1a1bf04be
00000001 a1bf04c0 00000020 00000400 00000008 8000115f84f09f12 0000 000002a1a1bf04c0
00000001 a1bf04c2 00000020 00000401 00000008 0c1c1205690d81f8 0000 000002a1a1bf04c2
00000001 a1bf04c3 00000020 00000401 00000007 001fa2420000624d 0000 000002a1a1bf04c3
00000001 a1bf04c5 00000020 00000402 00000008 0c2a5a95b99d0286 0000 000002a1a1bf04c5
00000001 a1bf04c7 00000020 00000402 00000007 001faa6000003c49 0000 000002a1a1bf04c7
00000001 a1bf04c8 00000020 00000403 00000008 0c1c0c06840e02d2 0000 000002a1a1bf04c8
00000001 a1bf04ca 00000020 00000403 00000007 001fad4200006c5d 0000 000002a1a1bf04ca
00000001 a1bf04cc 00000020 00000404 00000008 0c1c0882800b82d8 0000 000002a1a1bf04cc
00000001 a1bf04cd 00000020 00000404 00000007 001fad8200009cd1 0000 000002a1a1bf04cd
00000001 a1bf04cf 00000020 00000405 00000008 0c1c0f04850cc2de 0000 000002a1a1bf04cf
00000001 a1bf04d0 00000020 00000405 00000007 001fada20000766f 0000 000002a1a1bf04d0
00000001 a1bf04d2 00000020 00000406 00000008 0c1bd80c4e13831a 0000 000002a1a1bf04d2
00000001 a1bf04d3 00000020 00000406 00000007 001faf800000505b 0000 000002a1a1bf04d3
00000001 a1bf04d5 00000020 00000407 00000008 0c23d51049974330 0000 000002a1a1bf04d5
00000001 a1bf04d6 00000020 00000407 00000007 001fb02000004873 0000 000002a1a1bf04d6
00000001 a1bf04d8 00000020 00000408 00000008 0c1c0a8490cc44ba 0000 000002a1a1bf04d8
00000001 a1bf04da 00000020 00000408 00000007 001fb762000088bf 0000 000002a1a1bf04da
00000001 a1bf04db 00000020 00000409 00000008 0c1c0603a0cbc4c0 0000 000002a1a1bf04db
00000001 a1bf04df 00000020 00000409 00000007 001fb76000008ee5 0000 000002a1a1bf04df
00000001 a1bf04e0 00000020 0000040a 00000008 0c23f70c5b9544cc 0000 000002a1a1bf04e0
00000001 a1bf04e2 00000020 0000040a 00000007 001fb7820000565f 0000 000002a1a1bf04e2
00000001 a1bf04e3 00000020 0000040b 00000008 0c1bf3049b4cc502 0000 000002a1a1bf04e3
00000001 a1bf04e5 00000020 0000040b 00000007 001fb82200007eab 0000 000002a1a1bf04e5
And finishes up with this:
Total Size: 0x00000000085ae0a8
Where that number in decimal is 140,173,480. That's about what I expected.
Update:
In order to get closer to what you asked, I took the code in the ReadStruct method and used it to create a corresponding WriteStruct method:
private static void WriteStruct<T>(this BinaryWriter writer, T obj) where T : struct {
var len = Marshal.SizeOf(typeof(T));
var buffer = new byte[len];
GCHandle handle = default(GCHandle);
try {
handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
Marshal.StructureToPtr(obj, handle.AddrOfPinnedObject(), false);
} finally {
if (handle.IsAllocated)
handle.Free();
}
writer.Write(buffer);
}
With that, I can also modify my original code to read all the data, and write selective parts out to another file. In the code below, I read in the "Chunks" until the timestamp on the chunks is divisible by 10,000. Once that happens, I create a new CanHeader structure (I'm not exactly sure what should go there - but you should be). Then I create an output FileStream (i.e., a file to write to) and a BinaryWriter. I write the header to the FileSteam, and then I write the next 5000 chunks I read to that file. In your case, you can use the data in the chunk stream to decide what you want to do:
using (var readStream = new FileStream("Klassifikation_only_Sensor1_01.dr2", FileMode.Open, FileAccess.Read)) {
using (var reader = new BinaryReader(readStream)) {
var headerTuple = reader.ReadStruct<CanHeader>();
Console.WriteLine($"[Header] TimeStampFrequency: {headerTuple.Item1.TimeStampFrequency:x016} TimeStamp: {headerTuple.Item1.TimeStamp:x016}"); ;
bool stillWorking;
UInt64 totalSize = 0L;
UInt64 recordCount = 0L;
var chunkSize = (UInt64)Marshal.SizeOf(typeof(CanChunk));
var chunksWritten = 0;
FileStream writeStream = null;
BinaryWriter writer = null;
var writingChucks = false;
var allDone = false;
try {
do {
var chunkTuple = reader.ReadStruct<CanChunk>();
stillWorking = chunkTuple.Item2;
if (stillWorking) {
var chunk = chunkTuple.Item1;
if (!writingChucks && chunk.CanTime % 10_000 == 0) {
writingChucks = true;
var writeHeader = new CanHeader {
TimeStamp = chunk.TimeStamp,
TimeStampFrequency = headerTuple.Item1.TimeStampFrequency
};
writeStream = new FileStream("Output.dr2", FileMode.Create, FileAccess.Write);
writer = new BinaryWriter(writeStream);
writer.WriteStruct(writeHeader);
}
if (writingChucks && !allDone) {
writer.WriteStruct(chunk);
++chunksWritten;
if (chunksWritten >= 5000) {
allDone = true;
}
}
totalSize += chunkSize;
++recordCount;
}
} while (stillWorking);
} finally {
writer?.Dispose();
writeStream?.Dispose();
}
Console.WriteLine($"Total Size: 0x{totalSize:x016} Record Count: {recordCount} Records Written: {chunksWritten}");
}
}
}
When I'm finished, I can see that 5000 records are written to the file (it's 200,016 bytes long - 5000 40-byte records prefaced with a 16 byte header), and that the first record's CanTime is 0xa3a130d0 (or 2,745,250,000 - i.e., divisible by 10,000). Everything is I expect.
