How to Read Multiple Files Simultaneously with TPL

Question

I want to create a Winforms app to read 48 text files, each with only one line of comma separated values and populate a MySQL database and DataGridView with the results from each.

From Trawling SO and Google and months of trial and error the best I've been able to come up with is the following:

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
    BackgroundWorker1.RunWorkerAsync()
End Sub

Private Sub BackgroundWorker1_DoWork(sender As Object, e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork
    UpdateMcs()
End Sub

Private Sub BackgroundWorker1_RunWorkerCompleted(sender As Object, e As System.ComponentModel.RunWorkerCompletedEventArgs) Handles BackgroundWorker1.RunWorkerCompleted
    Me.MainTableAdapter.Fill(Me.MyDataSet.main)
    If stopPolling = False Then BackgroundWorker1.RunWorkerAsync()
End Sub

Private Sub UpdateMcs()
    Dim tasks As New List(Of Task)
    For Each row As DataRow In MyDataSet.main
        If row("model") = "A" Then
            tasks.Add(Task.Factory.StartNew(Sub()
                                                Try
                                                    Dim Path As String = "\\" & row("server_ip") & "\mc_data\mc_" & Format(row("mc_num"), "00") & ".txt"
                                                    Dim lines() As String = File.ReadAllLines(Path)
                                                    Dim data As String = lines(0)
                                                    Dim state As New stateTableAdapter
                                                    Try
                                                        state.UpdateByIP(data, row("ip"))
                                                        Console.WriteLine("{0}: {1}", row("ip"), "Success")
                                                    Catch ex As Exception
                                                        Console.WriteLine("{0}: {1}", row("ip"), ex.Message)
                                                    End Try
                                                Catch ex As Exception
                                                    Console.WriteLine("{0}: {1}", row("ip"), ex.Message)
                                                End Try

                                            End Sub))
        End If
    Next
    Task.WaitAll(tasks.ToArray())
End Sub

What I'd like to do though is to have each machine update the DB and refresh UI as it is completed and not wait for the rest to complete.

I've tried doing this by spawning individual threads for each machine but it has resulted in very buggy behaviour and locked up UI etc. So not very successful.

I should also note that I have another 39 machines that will be polled for data using an API that will also need to be run individually.

Is there a more efficient way to tackle this problem?

One of the main issues is that if a machine (or a server hosting data file) is offline for some reason the time to establish that fact vs the time to poll a machine that is online could be 500ms vs 10secs so having all online machines waiting for the offline one means that the DB update frequency is severely limited by the worst performing poll.

Any advice on better techniques to use and resources for studying would be very welcome!!

usr · Answer 1 · 2015-02-18T15:33:01.683

1

This is a great case for async IO. The number of threads that you need for sync IO is causing problems so let's use async IO to remove them.

Looks like almost all time is spent in ReadAllLines so make that one async.

It could look roughly like this in C#:

tasks.Add(Task.Run(async () => {
 var data = await File.ReadAllLinesAsync(...);
 //rest of code here
});

That would be pretty much all the change you need to make.

Note, that Task.WaitAll will block the thread it runs on. If you want to avoid that use await here as well (await Task.WhenAll(tasks)).

edited Feb 18 '15 at 15:33

answered Feb 18 '15 at 12:13

usr

168,620
35
240
369

You have missed `async` for lambda. – Hamlet Hakobyan Feb 18 '15 at 13:06
Does using `Task.Run` in this situation "block" a thread pool thread? – Cameron Feb 18 '15 at 16:18
@Cameron for a very short amount of time (for the amount of time it takes to reach the first await which should be near zero). You could leave it out but it seemed like a safer choice since he is running this on the UI thread I think. – usr Feb 18 '15 at 16:31
Is `ReadAllLinesAsync` a custom function? I can't find any mention of it...? – doovers Feb 18 '15 at 23:09
I've attempted to write my own `ReadAllLinesAsync` function but I can't get it to work. Also, should I use the `BackgroundWorker.ProcessChanged` event to fire the DB & UI updates as each file is read or is there a better solution? – doovers Feb 18 '15 at 23:46
Surprisingly, ReadAllLinesAsync does not exist. It was a guess :) Yeah, you'll have to write this on your own. http://stackoverflow.com/a/13168006/122718. You no longer need BackgroundWorker with await. Simply update the UI directly. If this sounds suspicious to you make yourself more familiar with await. – usr Feb 19 '15 at 10:21

How to Read Multiple Files Simultaneously with TPL

1 Answers1