0

I'm fetching more than 10 millions of records from database and writing to a text file. It takes hours of time to complete this operation. Is there any option to use TPL features here?

It would be great if someone could get me started implementing this with the TPL.

using (FileStream fStream = new FileStream("d:\\file.txt", FileMode.OpenOrCreate,     FileAccess.ReadWrite))
{
    BufferedStream bStream = new BufferedStream(fStream);
    TextWriter writer = new StreamWriter(bStream);
    for (int i = 0; i < 100000000; i++)
    {
        writer.WriteLine(i);
    }
    bStream.Flush();
    writer.Flush(); // empty buffer;
    fStream.Flush();
}
Adam
  • 15,537
  • 2
  • 42
  • 63
user1039583
  • 41
  • 8
  • 8
    What database is it? In all probability the database has tools to export data much more efficiently / faster than using C#. – Alvaro Rodriguez Aug 14 '12 at 14:01
  • 3
    What part of the operation is taking the most time? The query, the db data transfer, or writing to disk? – D'Arcy Rittich Aug 14 '12 at 14:02
  • 1
    unless you have a hard-drive that is capable of writing thousands of lines at once, writing to the file from several different threads will not make it any faster (on the contrary). The database side is a whole different story, but you haven't shown the code for that. – Adam Aug 14 '12 at 14:08
  • Hi, Actually, Writing to file takes more time than others. So Just thought i could use TPL to write it fastly. – user1039583 Aug 14 '12 at 14:12
  • I tried this code to implement but it doesnt write full content to the fileTask.Factory.FromAsync( fileStream.BeginWrite, fileStream.EndWrite, buf, 0, buf.Length, null). ContinueWith((result) => fileStream.Close()); – user1039583 Aug 14 '12 at 14:15
  • 1
    that's the point! your hard drive is **slower** than the processor, so whatever you do, you won't be able to get faster than your hard drive is! – Adam Aug 14 '12 at 14:16
  • 1
    however, it takes about 2 minutes for me to write those 100000000 lines to disk. how can it take hours for you? post your real code. – Adam Aug 14 '12 at 14:20
  • 1
    @user1039583 if you have profiled your application and you know the database side is not the problem, then show us what is really going on in that loop. it doesn't take that long to write those numbers to the disk, I just tried it! – Adam Aug 14 '12 at 14:26
  • Get rid of the BufferedStream. FileStreams are already buffered. Also, see http://stackoverflow.com/questions/1862982/c-sharp-filestream-optimal-buffer-size-for-writing-large-files for FileStream buffer size info. – JamieSee Aug 14 '12 at 16:04

1 Answers1

0

The codesample just writes integers to a file, and does no database access. If that's not quick something is wrong with your hardware. Your most likely bottleneck is fetching the data from the database. Either a slow query or slow connection. If you specify the databasetype and show the query we could zoom in on this...

IvoTops
  • 3,463
  • 17
  • 18