5

I know that it is not recommended to use Entity Framework for bulk inserting since it adds them one by one. (Example 1, example 2.)

How about attaching one object that has a large graph underneath it, e.g. 100Ks of objects? - Specifically:

  • What are the (time/memory) performance considerations for using EF5 to do this?
  • Are there any best practices for storing large graphs?
  • Are relational databases even a good idea for storing large graphs?

A bit of background:

  • The graph is created using EF5 POCOs, then attached at the root and saved. The graph is always new data, never updates. The DB we use is SQL Server 2012.

  • The actual creation process can take tens of seconds for large cases (100Ks of objects), so if the attach/save process takes 10% of that time, it won't have a significant affect on the overall time performance.

Community
  • 1
  • 1
Danny Varod
  • 17,324
  • 5
  • 69
  • 111
  • It not necessarily to add entities one by one. In my practice, I've divided my object graph to independent (almost) parts and loads it step-by-step to preserve integrity and not to get `OutOfMemoryException` or just hours of hopeless waiting. – Fedor Jun 27 '13 at 20:01
  • @Fyodor I didn't get an out of memory exception and it didn't take hours. – Danny Varod Jun 29 '13 at 09:38
  • Then you are lucky and your graph is more handy, than my was. – Fedor Jun 29 '13 at 10:54

1 Answers1

4

As no one have answered for some reason, even commonly known here EF expert, then I'll try to ask.

Recently, I've dealed with complex model with large amount of cycle referencing and numbers of relationsbetween entities. And large amount of data, yes, just as yours or bigger. I've tried many things and none were convenient for me, so I've ended with using Fastest Way of Inserting in Entity Framework. It helped me a lot a year ago and you haven't mention it in your question post somehow.

But if you haven't got many complex relations between entities and have large amount of completely new data to insert just one time, then the fastest way as you have already mentioned is SqlBulkCopy because it is extremely fast and simple in this case.
Using CodeFirst approach, for example, you could firstly create database structure and then simply insert your data.

Hope it helps! No guess if using NoSQL storage will be fast or convinent in your case as I've never used those solutions. May be experienced colleagues will add some about it.

Community
  • 1
  • 1
Fedor
  • 1,548
  • 3
  • 28
  • 38
  • So the more entities added in one operation, the faster, as long as the operation doesn't take up too much memory. Divide by object type and insert all object of that type using SqlBulkCopy is the fastest? – Danny Varod Jun 29 '13 at 09:46
  • Yes, this is the fastest way of doing this. It will take just seconds to insert all those entries. – Fedor Jun 29 '13 at 10:45