Is my dictionary size normal?

Question

I have a 150mb file. Each line is made up of the same format eg/

I,h,q,q,3,A,5,Q,3,[,5,Q,8,c,3,N,3,E,4,F,4,g,4,I,V,9000,0000001-100,G9999999990001800000000000001,G9999999990000001100PDNELKKMMCNELRQNWJ010, , , , , , ,D,Z,

I have a Dictionary<string, List<string>>

It is populated by opening the file, reading each line, taking elements from the line and adding it to the dictionary, then the file is closed.

StreamReader s = File.OpenText(file);
 string lineData = null;
 while ((lineData = s.ReadLine()) != null)
 {
   var elements = lineData.Split(',');
   var compareElements = elements.Take(24);
   FileData.Add(elements[27], new List<string>(compareElements));

  }
  s.Close();

Using the method in this answer I calculated my dictionary to be 600mb. That's 4 times what the file is.

Does that sound correct?

Answer 1

Apart from the fact that the method is not very reliable, in your case there is an even greater overhead. Did you notice that each iteration of your loop creates a new instance of the elements array, the lineData string, and the elements.Take also has some internal variables that get created on each call? Since you probably have enough RAM, the .NET garbage collector doesn't bother collecting them, so when you measure the TotalMemory before and after the loop, you also measure all these variables, not only your dictionary, although it might be the only thing that remains in scope afterwards.

Answer 2

Yes, because you are turning chars into string pointers, which are 4 or 8 bytes each.

Answer 3

I assume your file is encoded UTF-8 and contains mostly ASCII. strings in C# are UTF-16 though, so that explains most of the size difference right there (factor of 2). There is also some overhead for data structures of course.

Answer 4

Most of these entities take only a single character, yet you are storing them as strings. The reference pointer to those string alone is going to take at least twice as much space (in case of UTF8 likely 4-8 times as much). Then there is the overhead of keeping a hash table structured for the dictionary.

The List<> in itself should be really efficient storage wise (it uses an array internally)

Room for improvement:

• you could use List<char> or char[] instead of List<string> if you know that the fields will fit
• you could use struct Field { char a,b/*,...*/; } and List instead of List if you need more than 1 character per field

• You could forgo the eager field extraction [<-- recommended]:

   var dict = File.ReadAllLines(file)
        .ToDictionary(line => line.Split(',')[27]);
  

  This gives you the opportunity to access the compareElements on demand:

   string[] compareElements = dicts["key27"].Split(',')/*.Take(24).ToArray()*/;
  

  This is a classic example of runtime/storage cost trade-off

Edit an obvious hybrid would be:

struct AllCompareElements
{
     public char field1, field2, ... field24;
     // perhaps:
     public char[2] field13; // for the exceptional field that is longer than 1 character
}

Happily employ Resharper to implement Equals, GetHashCode, IEquatable<AllCompareElements>, IComparable<AllCompareElements>

Answer 5

If your file is encoded in ANSI or UTF-8 (but without special characters then size is same as ANSI) (each char 1 byte) and string - "Represents text as a series of Unicode characters." (Unicode = UTF-16, each char 4 bytes) it's 4 times more.

Answer 6

That's 600M was allocated by the operation of loading the file into the dictionary... Suggests it's an expensive operation, and it could be useful in guaging how effective any optimisation might be, but for how much memory the dictionary takes up, fairly useless.

I'd defer the splitting as suggested by sehe straight off.

Looks to me like you've preamturely optimised for speed and that's cost you big style on memory foot print.