1

Please assist. what is fastest way to read more than 3000 .CSV fil moes Data from a location to Sql table through .net c#

I am using Ado.net .
size of each file is around 120 kb file contains , separated data

Please assist me if any have idea about this

Laxminarayan Charan
  • 79
  • 1
  • 10

2 Answers2

4

reading the files can be done in Parallel, something like this: 

var files = someFolder.GetFiles("?.csv"); // Get all csv files
Parallel.Foreach(files, file => {

  // insert contents into table
});

This is however only half the answer. Some also need to tell you the optimal way to do the inserts into the SQL database. You aren't stating if you're using ADO or EntityFramework or some other mechanism to connect to the server, which is kind of essential to speed and for us to give you a good example.

Pedro G. Dias
  • 3,162
  • 1
  • 18
  • 30
  • Will it be fast, also let me know should I use data table or directly insert , seperatd file into table using sqlbulkvopy – Laxminarayan Charan Apr 13 '16 at 06:46
  • The reading of the files will be fast, yes. I have a project where I am processing every single word in the English dictionary spread across a ton of text files, and the file part takes around 15ms (0.015 seconds), granted there are only 26 text files to read. – Pedro G. Dias Apr 13 '16 at 07:49
  • please paste logical part of code,,,I need to implement urgently...please help – Laxminarayan Charan Apr 13 '16 at 07:57
  • Dude, anybody who needs to implement code urgently will actually follow advice they are given. You asked this question 2 hours ago, when even a brute-force naive approach would have got the job done. What are your motives? Why are you here? And why are you harassing people to do your job for you? This answer _IS_ the logical part of the code. How much more do you need? – paddy Apr 13 '16 at 08:09
0
  1. Separate problems: reading files and pushing data to DB are different concerns,
  2. Load files into memory buffer as batches of e.g. no more than 200k total rows,
  3. Use SqlBulkCopy class to insert batches of rows from your buffer into the table (by batches of e.g. 20k rows),
  4. Use async to do 2. and 3. jobs asyncronously and as parallel - read .csv files when pushing rows to DB, because it uses different resources (network and file system).
pkuderov
  • 3,501
  • 2
  • 28
  • 46