-3

Which technique is preferred to use when the requirement is to store large amounts of data (reference type) and frequent lookup is being performed?

bytecode77
  • 14,163
  • 30
  • 110
  • 141
Abhinaw Kaushik
  • 607
  • 6
  • 18
  • 2
    Define “large volume of data”'. Kilobytes? Gigabytes? Petabytes? – Dour High Arch Aug 03 '16 at 19:13
  • 1
    How are you looking it up? – EJoshuaS - Stand with Ukraine Aug 03 '16 at 19:13
  • 2
    There's not enough information here to answer the question. If your data can't fit in memory, then neither one is appropriate. – recursive Aug 03 '16 at 19:15
  • @DourHighArch *Bites? Nibbles? Bytes? Kilobytes? Megabytes? Gigabytes? Terrabytes? Petabytes? Exabytes? Zettabytes? Yottabytes? – BanForFun Aug 03 '16 at 19:21
  • Neither. Whenever you have large amount of data use a database that is designed to store large amount of data. – jdweng Aug 03 '16 at 19:53
  • @jdweng Of course, the question could just as easily be about processing the results of a large database query. At the end of the day the quesiton simply lacks the information to be answerable. – Servy Aug 03 '16 at 20:19
  • Requirement here is to store millions of records from a table which contains 10-15 columns and 90% columns are nvarchar and on an average 20 characters will be stored per column; and lookup will be done on a key – Abhinaw Kaushik Aug 04 '16 at 19:33

2 Answers2

1

A Dictionary is hashtable-based and key lookup time is O(1), whereas a List's lookup time is O(n). For large data volumes, a dictionary will be much faster. There is a great deal of information on this out on Google and SO already, such as C# : Why is dictionary so much faster than list?

Community
  • 1
  • 1
DVK
  • 2,726
  • 1
  • 17
  • 20
  • It's O(1) if (and only if) you know the key of what you're looking for. For other kind of lookup it performs much worse than that; for example, it's O(n) if you have to search for a value. – EJoshuaS - Stand with Ukraine Aug 03 '16 at 19:15
  • 1
    I assumed key lookups. Otherwise there is no reason for using a dictionary. – DVK Aug 03 '16 at 19:16
  • 1
    Dictionary was my first thought too actually but I think he has to clarify his post to know for sure if that's an appropriate solution in this case - there's not enough information on how he plans to do the lookup or exactly how much data there is. It may (or may not) be the case that he knows the key in advance, for example, he doesn't share enough information to know for sure. – EJoshuaS - Stand with Ukraine Aug 03 '16 at 19:18
0

If you're concerned that a Dictionary will not suit you for reasons of performance, then maybe a Dictionary is not the right way to store data.

Since you didn't provide much information, I will try my best by guessing you have huge amounts of data which are possibly persisted. In this case, a database would solve the problem, because SQL databases are far more scalable than arrays or lists of objects.

You can even use Entity Framework to access your database structure as objects. A table in your database will be mapped to a class with properties for each column. This will yield the performance that raw SQL gives you, except if you are building more complex queries.

But it's definitely worth looking into.

bytecode77
  • 14,163
  • 30
  • 110
  • 141