0

We are very successfully using EF 5.0 for our real time server as well as from our internal websites. Now I need to create a utility that parses the data history to create a new table, which will be done using a copy of the production database used for data mining. Given that EF is transaction based, is there a good way to create a very large table where the table may have > 1M rows? My current thinking is no, and that the way to do this is to perhaps read the data with EF, but create a CSV file that is then bulk loaded, which I successfully do in some other situations. I'm not necessarily looking for the most efficient way, but I cannot imagine that EF or SQL would do well to add > 1M records with a single transaction. I know I could batch them in 1000 record chunks but that is not especially appealing. EF is said to be MSFT's principal data access technology going forward but they need to support this sort of scenario as part of that plan. Any ideas and insight appreciated. Thx.

Dave
  • 1,822
  • 2
  • 27
  • 36
  • Of course it's just a marketing slogan. Everybody knows a slogan is only partly true. EF is great for the majority of OLTP applications. But your case is not a common application use case, not at all. It demands other technologies, e.g. integration services. – Gert Arnold Dec 30 '13 at 21:32
  • I've heard it presented as much more than that, and imo it's both conceivable and desirable that M$FT wrapper these other scenarios, but I guess that is for a discussion elsewhere, and not a StackOverflow question... – Dave Dec 31 '13 at 22:57

1 Answers1

1

EF is not geared for bulk operations (see Efficient way to do bulk insert/update with Entity Framework).

Instead of using a CSV to bulk-load, perhaps you want to look into the SQL Server Bulk Copy API

Community
  • 1
  • 1
driis
  • 161,458
  • 45
  • 265
  • 341
  • Thx. Undoubtedly the answer, and I will eventually probably mark as such, but would love to see how this problem would be addressed in relation to MSFT's claim this is the way to go for new applications. http://msdn.microsoft.com/en-us/data/ef.aspx – Dave Dec 30 '13 at 19:10