How can I send multiple xml/HTTP requests at once and grab the resulting string? Vb.net

Question

So I have an IP Camera that has information I want (date, firmware version, etc) and each piece of information is grabbed by going to a Http.GetResponse() from an http URL, and getting an XML responding string. For example, "http://" & ip & "/System/Audio/Channels" gives you a string about its audio channels.

I need a lot of information, but the device doesn't have a URL to list all the items I need, so I repeat this process for each device with x number of item specific URLs. Here's a code snippet of using just one URL:

  Dim RespStr As String = "http://" & ip & "/System/Audio/Channels"

   'Open Request...
    Dim HttpReq As Net.HttpWebRequest = Nothing
    Try

        'create request object
        HttpReq = CType(Net.WebRequest.Create(ReqStr), Net.HttpWebRequest)

        'set the username and password
        HttpReq.Credentials = New Net.NetworkCredential(camera.Username, camera.Password)

        'set method
        HttpReq.Method = "GET"

        'disable expect-100-continue
        HttpReq.ServicePoint.Expect100Continue = False

        'timeout
        HttpReq.Timeout = 1000 * 2

    Catch ex As Exception

    End Try

    'Open Response...
    Dim HttpResp As Net.HttpWebResponse = Nothing
    Dim HttpStatus As Net.HttpStatusCode = Nothing
           Try

            'get the response
            HttpResp = CType(HttpReq.GetResponse(), Net.HttpWebResponse)

            'verify the response
            HttpStatus = HttpResp.StatusCode
            If HttpStatus <> Net.HttpStatusCode.OK Then

               'error
            End If
      'do stuff to process xml string



        Catch ex As Exception

         End Try

Obviously, after the 10th time in a loop for a specific URL, you start to get slow and repetitive.

Is there a way to tell vb.net in a quicker way to go to url1, url2, url3 (all similar to the example I mention above) and concatenate all the string responses in one network attempt? Possibly even at once since it's the same IP Address? Then I can just process it on my end versus over the network.

Have a look at some of the Parallel libraries available. http://msdn.microsoft.com/en-us/library/dd460705(v=vs.110).aspx You could run multiple requests simultaneously and store each value to process after they are all complete. — xDaevax, Aug 20 '14 at 15:20
I like the solution! Do you know if there are any special actions that I might have to do because it's going over the network? Or does windows do all the queuing? — Kat, Aug 20 '14 at 15:23

score 0 · Answer 1 · edited May 23 '17 at 11:49

By leveraging the .NET Framework's parallel libraries, you can speed up your process by having multiple similar tasks execute in parallel.

Documentation: http://msdn.microsoft.com/en-us/library/dd460705(v=vs.110).aspx

There aren't any special actions that need to be performed, but there are some considerations:

You may get blocked if you make too many requests too frequently to the destination (introduce some rate-limiting).
Whichever machine is running this code will need to be able to support multiple threads.
You may want a re-try mechanism.
You are limited to a specific number of concurrent network requests in .NET see( http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit(v=vs.110).aspx) and (System.Net.ServicePointManager.DefaultConnectionLimit == 24 --> BUG?)

Here is a brief implementation:

Public Class Program

    Shared Sub Main()

        Dim urls As New ConcurrentQueue(Of String)
        urls.Enqueue("www.google.com")
        urls.Enqueue("www.yahoo.com")
        Dim myMethod As Action = Sub() 

            Dim localRequest As String
            Dim localResponse As String
            While urls.TryDequeue(localRequest)
                System.Threading.Thread.Sleep(100) 'Rate limiting, adjust to suit your needs
                localResponse = WebWorker.MakeRequest(localRequest)
                Console.WriteLine(localResponse.ToString())
                'Do something with localResponse
            End While

        End Sub
        Parallel.Invoke(New ParallelOptions() With {.MaxDegreeOfParallelism = 10 }, myMethod, myMethod, myMethod)
    End Sub

    'Do something with the responses
End Class

Public NotInheritable Class WebWorker
    Private Sub New()
    End Sub

    Public Shared Function MakeRequest(request As String) As String
        Dim response As New String()
        Dim status As Boolean
        Dim req As HttpWebRequest = Nothing
        Dim resp As HttpWebResponse = Nothing
        Dim maxIterations As Integer = 2
        Dim currentAttempt As Integer = 0

        While currentAttempt < maxIterations AndAlso status = False
            Try
                req = DirectCast(HttpWebRequest.Create(New Uri(request)), HttpWebRequest)

                Using resp = DirectCast(req.GetResponse(), HttpWebResponse)
                    If resp.StatusCode = HttpStatusCode.OK Then
                        status = True
                    Else
                        currentAttempt += 1

                    End If
                    ' end using
                End Using
            Catch ex As Exception
                currentAttempt += 1
            End Try
            ' end try/catch
        End While
        ' end while
        Return response
    End Function

End Class

How can I send multiple xml/HTTP requests at once and grab the resulting string? Vb.net

1 Answers1