3

I have a loop that processes data, row by row from a SqlDataReader. It works dandy. Now I want to add parallelism to it. Ideally I want to read a row, toss it to a thread, read another row, etc. I'd like a configurable number of threads (say 10) so as one opens up (completes its task), another is allowed to start.

Is there a built in way to do this, or should I handle it myself? I looked at PLINQ and Parallel, but I have trouble getting my head around it i think.

Mike Perrenoud
  • 66,820
  • 29
  • 157
  • 232
Jonesopolis
  • 25,034
  • 12
  • 68
  • 112
  • 1
    Here is a similar question asking the same thing: [Is there a way to use the Task Parallel Library(TPL) with SQLDataReader?](http://stackoverflow.com/questions/3096727/is-there-a-way-to-use-the-task-parallel-librarytpl-with-sqldatareader). Be sure to look at both the accepted answer and the other answer. – Scott Chamberlain Jul 12 '13 at 15:15

1 Answers1

2

Since you're using a DataReader, it's forward only and one row at a time in memory, so you're going to need to handle this on your own. You're going to need to marshal the row in memory into some data structure and pass that on to the thread and then issue Read() again to move next.

You'll have to determine how you're going to pause when you have too many threads open already, but that's a whole other discussion and if you end up having questions about that then post what you have and we'll gladly help!

Mike Perrenoud
  • 66,820
  • 29
  • 157
  • 232
  • Thanks Michael. In your opinion is there another/better option than DataReader? There's just too many rows to read it all into memory at once. – Jonesopolis Jul 12 '13 at 15:11
  • Also, do you have any idea where to find an example of implementing this? (Mostly the correct terminology for a google search) – Jonesopolis Jul 12 '13 at 15:14
  • 2
    @Jonesy, I don't think there really is a better way. I think you're doing it right. And further, there probably isn't anything really good to search for because this is a pretty custom implementation - I would just write it. In short, take the row in memory and put it into a new object and then queue that object for the background threads. I would recommending using a `BackgroundWorker` and the `ConcurrentQueue`, and I wouldn't worry about **pausing** unless you actually run into a problem. The `ConcurrentQueue` is thread safe so you can read/write from different threads safely. – Mike Perrenoud Jul 12 '13 at 15:20
  • 2
    The link I added to the original question has a example of some code on how to do it. – Scott Chamberlain Jul 12 '13 at 15:24