2

I have an application where I am persisting a large amount of binary data fairly quickly. I currently have two different data channels that can grow in size independently, so I store this using two files. However, these two files are closely coupled to each other (ID references that only make sense in the context of the other file), so it would be preferable, for the purposes of transport and proper linkage, to store/transport them as a single file.

Is there some standard way, at either the OS level or within the .NET BCL to construct two FileStream objects that can grow independently within the same underlying file?

I've considered implementing some sort of chunking/paging scheme, but it strikes me that this is basically duplicating what the file system already does under the hood, so I'd rather let the file system manage that, if it's possible.

Another alternative is to use a post-processing tool to combine the two files after recording completes. This is manageable, but not ideal.

I've experimented with some lightweight database options, as well, but they've proven to be too slow for what I'm doing.

An ideal solution to this problem would scale to more than two data streams in the same file, though two would be sufficient for my current purposes.

Dan Bryant
  • 27,329
  • 4
  • 56
  • 102
  • Do you need any sort of structure in the final file, I mean once you close the two streams, do you need anything but your program, which you can change to suit the file format, to open the file? Say you end up with chunk 1 + chunk 2, could a valid storage be something akin to a zip file, or do you need them to be byte-to-byte in the final file, first chunk 1 then chunk 2. – Lasse V. Karlsen Feb 09 '15 at 18:15
  • How should that work? Where are new bytes written to the single file? Some paging? – DrKoch Feb 09 '15 at 18:16
  • There is [COM Structured Storage](http://en.wikipedia.org/wiki/COM_Structured_Storage) - I use it in one of my products for storing multiple inter-related streams of data into a single physical file. It's somewhat dated technology, though, and could be a bit awkward to use from C#. I am not sure if there is a .NET-based equivalent. – 500 - Internal Server Error Feb 09 '15 at 18:17
  • @LasseV.Karlsen, I'm flexible on the structure and only my program needs to open the file. Both files are currently formatted using Protobuf.net to serialize a series of binary records, preceded by a header record with some version information. – Dan Bryant Feb 09 '15 at 18:18
  • I too was going to suggest looking at structured storage, and share the same thoughts that it is both dated and may not have a good .NET implementation. I have looked at a few, but haven't found any implementation that looks good. – Lasse V. Karlsen Feb 09 '15 at 18:27
  • If you are using NTFS [Alternate Data Streams](https://en.wikipedia.org/wiki/NTFS#Alternate_data_streams_.28ADS.29) sounds like a (bad) possibility. ([FAQ](http://www.heysoft.de/en/information/ntfs-ads.php?lang=EN)) – HABO Feb 09 '15 at 18:27

2 Answers2

4

I'd just go for a zip file. Ionic dotnetzip works pretty nicely with streams so that you don't end up having to load them into memory (we use this technique to zip up huge payloads without bringing down the server):

using(var z = new ZipFile())
{
    z.AddEntry("file1", (name, stream) => 
    {
        someSourceStream1.CopyTo(stream);
    });
    z.AddEntry("file2", (name, stream) => 
    {
        someSourceStream2.CopyTo(stream);
    });
    z.Save(@"some\path.zip");
    //or
    //z.Save(someOutputStream);
}
spender
  • 117,338
  • 33
  • 229
  • 351
1

You can use NTFS alternate data streams (NTFS Alternate Data Streams - .NET) if you just want it to look like single file.

Note that many tools ignore alternate data streams during file operations, so be careful if you need to move such file around.

Community
  • 1
  • 1
Alexei Levenkov
  • 98,904
  • 14
  • 127
  • 179