0

I am making a combinations generator. For small amounts of elements it's not a problem for the computer that the data is getting stored in the RAM memory instead in a file. But when the number of elements gets bigger, my computer runs out of memory (the exception OutOfMemoryException occurs). The combinations are numbers stored in Lists, that are currently getting stored in an another List.

But this only the first step- the generator work right. I want the data to be stored in file, from where a different program will be able to extract the combinations it needs. Mostly, I need to store the data in a separate file, because the generator will have to be able to create more and more and bigger combinations in the future. The computer will have to read certain parts of the data, without coping all of it in the temporary memory, because this is impossible.

I don't want to turn the data into text and when needed to convert the text back into data. I think this will make things slower because of the conversions. I want the lists to be stored into a custom made file, from where the program can directly extract the data without any converting.

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
AlexSavAlexandrov
  • 858
  • 2
  • 13
  • 34
  • 1
    You need to store the data in binary format, or use a database. – Amiram Korach Jul 26 '12 at 14:54
  • 1
    This is good background information, but there does not appear to be a question anywhere. Also, is there some specific problem you are having with code? If so, please post what you have tried so far. – JYelton Jul 26 '12 at 14:54

4 Answers4

3

There is a ton of options available I will briefly describe a few.

  • Use a database. From your description this does not look like a good choices but it will be the most flexible to all clients relativity fast and efficient storage.

  • Use one of the .net serializers from your description binary serializer will be your best choice. The serlizers offer a lot of advantages relativity fast and baked into .net with built in support and very easy to use.

  • Use a custom binary format. This will be the fastest option especially if you combine it with a memory mapped file. However binary formats can be hard to use and easy to screw up.

rerun
  • 25,014
  • 6
  • 48
  • 78
3

If you really want to store your data in file, you can use BinaryFormatter class. It is probably the most efficient way of serializing data objects into binary stream.

But I wouldn't recommend you generating combinations in this way if only don't you need to store them at one time and load long time after that. It's better to use lazy-generation of combinations. One by one, completely generated without the need to "generate bigger combinations in future" (generate "the biggest" needed combinations one by one - you might want to change your generation algorithm a bit - there are plenty answers how to do that already)

Sasha
  • 8,537
  • 4
  • 49
  • 76
2

There's a good write up on how to serialize a List<> to a file at http://www.switchonthecode.com/tutorials/csharp-tutorial-serialize-objects-to-a-file

Dan Pichelman
  • 2,312
  • 2
  • 31
  • 42
  • This tutorial explains how to turn the data into a text file and that's what I DON'T want to do. A computer will create the file and a computer will read it, not a human user. I want to store the data into a non text file, from where the program can read the data without any converting. But does serialising allow to store the data into a non text files? This wasn't mentioned in the tutorial. – AlexSavAlexandrov Jul 26 '12 at 17:20
  • I just tried the code. It has a few minor bugs, but they're easily solved. The data file (despite the unfortunate choice of names) is definitely NOT a text file. – Dan Pichelman Jul 26 '12 at 17:37
  • Still, I'll have to figure out how the make the program read the file. This is not explained in the tutorial as well. I'll have to find a way to make the program read certain parts of the file, without reading the WHOLE data (otherwise, the computers memory will get filled to the limit far before it finishes reading the file). – AlexSavAlexandrov Jul 26 '12 at 17:41
0

You can use something as a persistent data structure instead, this will reduce the amount of memory needed by your app, without changint too much the current code. Have a look at this question:

Looking for a simple standalone persistent dictionary implementation in C#

there is a lot of resources doing that, in particular this answer seems to point to some really interesting links:

Community
  • 1
  • 1
Felice Pollano
  • 32,832
  • 9
  • 75
  • 115