Visual Basic - Issue with how the code is being Executed

Question

So I have a button that Scrapes Proxies from Public Websites and adds the proxies to a Listbox but there's an issue, It for some reason shows the Proxies in the Listbox once its Actually finished doing all the "For Each"'s.

It even doesn't show a Progress Bar until everything is Finished which is making it pointless.

Screenshot of what I mean:

(As you can see it only shows the Progress Bar when its finished rather then while its Scraping.)

Is there any workaround for this?

My Code:

Imports System.Text.RegularExpressions

Public Class Form1

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

        Button1.Enabled = False
        ScrapeProgress.Visible = True

        'Set the Time;
        Dim time = TimeOfDay.ToString("h:mm:ss tt")

        'Sources;
        Dim sources(122) As String
        sources(0) = "http://proxy-ip-list.com"

        sources(1) = "http://fineproxy.org/eng/?p=6#more-6"

        sources(2) = "http://aliveproxy.com/high-anonymity-proxy-list"
        sources(3) = "http://aliveproxy.com/anonymous-proxy-list/"
        sources(4) = "http://aliveproxy.com/transparent-proxy-list"
        sources(5) = "http://aliveproxy.com/socks5-list"
        sources(6) = "http://aliveproxy.com/socks4-list"
        sources(7) = "http://aliveproxy.com/fastest-proxies/"

        sources(8) = "http://atomintersoft.com/anonymous_proxy_list"
        sources(9) = "http://atomintersoft.com/high_anonymity_elite_proxy_list"
        sources(10) = "http://atomintersoft.com/proxy_list_domain"
        sources(11) = "http://atomintersoft.com/proxy_list_port"
        sources(12) = "http://atomintersoft.com/transparent_proxy_list"
        sources(13) = "http://atomintersoft.com/products/alive-proxy/socks5-list"

        sources(14) = "http://best-proxy.com/english/search.php?search=anonymous-and-elite&country=any&type=anonymous-and-elite&port=any&ssl=any"
        sources(15) = "http://best-proxy.com/english/search.php?search=anonymous-and-elite&country=any&type=anonymous-and-elite&port=any&ssl=any&p=2"
        sources(16) = "http://best-proxy.com/english/search.php?search=anonymous-and-elite&country=any&type=anonymous-and-elite&port=any&ssl=any&p=3"

        sources(17) = "http://www.samair.ru/proxy/socks01.htm"
        sources(18) = "http://www.samair.ru/proxy/socks02.htm"
        sources(19) = "http://www.samair.ru/proxy/socks03.htm"
        sources(20) = "http://www.samair.ru/proxy/socks04.htm"
        sources(21) = "http://www.samair.ru/proxy/socks05.htm"

        sources(22) = "http://www.samair.ru/proxy/proxy-01.htm"
        sources(23) = "http://www.samair.ru/proxy/proxy-02.htm"
        sources(24) = "http://www.samair.ru/proxy/proxy-03.htm"
        sources(25) = "http://www.samair.ru/proxy/proxy-04.htm"
        sources(26) = "http://www.samair.ru/proxy/proxy-05.htm"
        sources(27) = "http://www.samair.ru/proxy/proxy-06.htm"
        sources(28) = "http://www.samair.ru/proxy/proxy-07.htm"
        sources(29) = "http://www.samair.ru/proxy/proxy-08.htm"
        sources(30) = "http://www.samair.ru/proxy/proxy-09.htm"
        sources(31) = "http://www.samair.ru/proxy/proxy-10.htm"
        sources(32) = "http://www.samair.ru/proxy/proxy-11.htm"
        sources(33) = "http://www.samair.ru/proxy/proxy-12.htm"
        sources(34) = "http://www.samair.ru/proxy/proxy-13.htm"
        sources(35) = "http://www.samair.ru/proxy/proxy-14.htm"
        sources(36) = "http://www.samair.ru/proxy/proxy-15.htm"
        sources(37) = "http://www.samair.ru/proxy/proxy-16.htm"
        sources(38) = "http://www.samair.ru/proxy/proxy-17.htm"
        sources(39) = "http://www.samair.ru/proxy/proxy-18.htm"
        sources(40) = "http://www.samair.ru/proxy/proxy-19.htm"
        sources(41) = "http://www.samair.ru/proxy/proxy-20.htm"
        sources(42) = "http://www.samair.ru/proxy/proxy-21.htm"
        sources(43) = "http://www.samair.ru/proxy/proxy-22.htm"
        sources(44) = "http://www.samair.ru/proxy/proxy-23.htm"
        sources(45) = "http://www.samair.ru/proxy/proxy-24.htm"
        sources(46) = "http://www.samair.ru/proxy/proxy-25.htm"
        sources(47) = "http://www.samair.ru/proxy/proxy-26.htm"
        sources(48) = "http://www.samair.ru/proxy/proxy-27.htm"
        sources(49) = "http://www.samair.ru/proxy/proxy-28.htm"
        sources(50) = "http://www.samair.ru/proxy/proxy-29.htm"
        sources(51) = "http://www.samair.ru/proxy/proxy-30.htm"

        sources(52) = "http://nntime.com/proxy-list-01.htm"
        sources(53) = "http://nntime.com/proxy-list-02.htm"
        sources(54) = "http://nntime.com/proxy-list-03.htm"
        sources(55) = "http://nntime.com/proxy-list-04.htm"
        sources(56) = "http://nntime.com/proxy-list-05.htm"
        sources(57) = "http://nntime.com/proxy-list-06.htm"
        sources(58) = "http://nntime.com/proxy-list-07.htm"
        sources(59) = "http://nntime.com/proxy-list-08.htm"
        sources(60) = "http://nntime.com/proxy-list-09.htm"
        sources(61) = "http://nntime.com/proxy-list-10.htm"
        sources(62) = "http://nntime.com/proxy-list-11.htm"
        sources(63) = "http://nntime.com/proxy-list-12.htm"
        sources(64) = "http://nntime.com/proxy-list-13.htm"
        sources(65) = "http://nntime.com/proxy-list-14.htm"
        sources(66) = "http://nntime.com/proxy-list-15.htm"
        sources(67) = "http://nntime.com/proxy-list-16.htm"
        sources(68) = "http://nntime.com/proxy-list-17.htm"
        sources(69) = "http://nntime.com/proxy-list-18.htm"
        sources(70) = "http://nntime.com/proxy-list-19.htm"
        sources(71) = "http://nntime.com/proxy-list-20.htm"
        sources(72) = "http://nntime.com/proxy-list-21.htm"
        sources(73) = "http://nntime.com/proxy-list-22.htm"
        sources(74) = "http://nntime.com/proxy-list-23.htm"
        sources(75) = "http://nntime.com/proxy-list-24.htm"
        sources(76) = "http://nntime.com/proxy-list-25.htm"
        sources(77) = "http://nntime.com/proxy-list-26.htm"
        sources(78) = "http://nntime.com/proxy-list-27.htm"
        sources(79) = "http://nntime.com/proxy-list-28.htm"
        sources(80) = "http://nntime.com/proxy-list-29.htm"
        sources(81) = "http://nntime.com/proxy-list-30.htm"

        sources(82) = "http://proxylistchecker.org/proxylists.php?t=&p=1"
        sources(83) = "http://proxylistchecker.org/proxylists.php?t=&p=2"
        sources(84) = "http://proxylistchecker.org/proxylists.php?t=&p=3"
        sources(85) = "http://proxylistchecker.org/proxylists.php?t=&p=4"
        sources(86) = "http://proxylistchecker.org/proxylists.php?t=&p=5"
        sources(87) = "http://proxylistchecker.org/proxylists.php?t=&p=6"
        sources(88) = "http://proxylistchecker.org/proxylists.php?t=&p=7"
        sources(89) = "http://proxylistchecker.org/proxylists.php?t=&p=8"
        sources(90) = "http://proxylistchecker.org/proxylists.php?t=&p=9"
        sources(91) = "http://proxylistchecker.org/proxylists.php?t=&p=10"
        sources(92) = "http://proxylistchecker.org/proxylists.php?t=&p=11"
        sources(93) = "http://proxylistchecker.org/proxylists.php?t=&p=12"
        sources(94) = "http://proxylistchecker.org/proxylists.php?t=&p=13"
        sources(95) = "http://proxylistchecker.org/proxylists.php?t=&p=14"
        sources(96) = "http://proxylistchecker.org/proxylists.php?t=&p=15"

        sources(97) = "http://txt.proxyspy.net/proxy.txt"

        sources(98) = "http://www.getproxy.jp/en/default/1"
        sources(99) = "http://www.getproxy.jp/en/default/2"
        sources(100) = "http://www.getproxy.jp/en/default/3"
        sources(101) = "http://www.getproxy.jp/en/default/4"
        sources(102) = "http://www.getproxy.jp/en/default/5"

        sources(103) = "http://www.ip-adress.com/proxy_list/?k=time&d=desc"

        sources(104) = "http://www.my-proxy.com/free-proxy-list.html"
        sources(105) = "http://www.my-proxy.com/free-proxy-list-2.html"
        sources(106) = "http://www.my-proxy.com/free-proxy-list-3.html"
        sources(107) = "http://www.my-proxy.com/free-proxy-list-4.html"
        sources(108) = "http://www.my-proxy.com/free-proxy-list-5.html"
        sources(109) = "http://www.my-proxy.com/free-proxy-list-6.html"
        sources(110) = "http://www.my-proxy.com/free-proxy-list-7.html"
        sources(111) = "http://www.my-proxy.com/free-proxy-list-8.html"
        sources(112) = "http://www.my-proxy.com/free-proxy-list-9.html"
        sources(113) = "http://www.my-proxy.com/free-proxy-list-10.html"

        sources(114) = "http://sslproxies24.blogspot.com/feeds/posts/default"
        sources(115) = "http://proxyserverlist-24.blogspot.com/feeds/posts/default"
        sources(116) = "http://newfreshproxies24.blogspot.com/feeds/posts/default"
        sources(117) = "http://socksproxylist24.blogspot.com/feeds/posts/default"
        sources(118) = "http://vip-socks24.blogspot.com/feeds/posts/default"
        sources(119) = "http://irc-proxies24.blogspot.com/feeds/posts/default"
        sources(120) = "http://www.socks24.org/feeds/posts/default"

        sources(121) = "http://www.getproxy.jp/en/"
        sources(122) = "http://www.getproxy.jp/en/fastest"

        'Add the Copyright Holder to the top of the List;
        ListBox1.Items.Add("Proxies Scraped using a Test Build Program by ShinyMK")

        'Foreach Source, Do the following Code;
        For Each element As String In sources

            'Set the Status Label to "Scraping";
            ToolStripStatusLabel3.Text = "Scraping"

            'Connect to the Proxy Source;
            Dim source As Net.HttpWebRequest = Net.WebRequest.Create(element)

            Try
                'Prepare the Response;
                Dim response As Net.HttpWebResponse = source.GetResponse

                'Load the HTML;
                Dim reader As IO.StreamReader = New IO.StreamReader(response.GetResponseStream())
                Dim html As String = reader.ReadToEnd

                'Regex;
                Dim expression As New Regex("[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}:[0-9]{1,4}")

                'Set the Matches variable to the Matched Sections of the HTML source;
                Dim matches As MatchCollection = expression.Matches(html)

                'Add the proxies to the ListBox;
                For Each proxy As Match In matches

                    'If the ListBox doesn't already contain the Proxy, Add It;
                    If Not ListBox1.Items.Contains(proxy.ToString) Then

                        'Add the Proxy;
                        ListBox1.Items.Add(proxy.ToString)

                        'Add 1 to the "Count" label's value;
                        ToolStripStatusLabel5.Text += 1

                    End If

                Next

                'Add the Time and Success message for each Successful Scraped Source;
                ListBox2.Items.Add("[" + time + "] Success: " + element)

            Catch ex As Exception

                ListBox2.Items.Add("[" + time + "] ERROR Scraping from " + element + " - " + ex.Message)

            End Try

        Next

        'Re-Enable the Button and set the Status Lable to "Idle";
        Button1.Enabled = True
        ToolStripStatusLabel3.Text = "Idle"

    End Sub

    Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click

        If ListBox1.Items.Count = (0) Then
            MessageBox.Show("No Proxies Scraped.", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error)
        Else
            'defining a streamwriter
            Dim S_W As IO.StreamWriter
            'converting listbox items to string
            Dim itms() As String = {ListBox1.Items.ToString}
            ''defining a savefiledialog
            Dim save As New SaveFileDialog
            Dim it As Integer
            save.FileName = "SSLPro.xyz " + TimeOfDay.ToString("h.mm.ss tt")
            save.Filter = "Text (*.txt)|*.txt|All Files (*.*)|*.*"
            save.CheckPathExists = True
            save.ShowDialog(Me)
            S_W = New IO.StreamWriter(save.FileName)
            For it = 0 To ListBox1.Items.Count - 1
                S_W.WriteLine(ListBox1.Items.Item(it))
            Next
            S_W.Close()
        End If

    End Sub

    Private Sub Button3_Click(sender As Object, e As EventArgs) Handles Button3.Click

        ListBox1.Items.Clear()
        ListBox2.Items.Clear()
        ToolStripStatusLabel5.Text = "0"

    End Sub

End Class

As a simplest solution - try to use `Application.DoEvents()` in a loop. Or better use asynchronous loading. — Paul, Apr 08 '16 at 09:55
Use [WebRequest.GetResponseAsync](https://msdn.microsoft.com/de-de/library/system.net.webrequest.getresponseasync%28v=vs.110%29.aspx) — Alex B., Apr 08 '16 at 09:58
@ShinyMK: https://stackoverflow.com/questions/202481/how-to-use-httpwebrequest-net-asynchronously — Paul, Apr 08 '16 at 09:58
@Paul. Oh, You guys need to realize that like, My whole app gets like Frozen as such until everythings done. When everythings done thats when it shows the Items in the Listbox and thats when it starts Doiing the "Visible = True" on the Progress bar. But I want it to make stuff visible and show the text right when the code gets read? You know what I mean? — , Apr 08 '16 at 10:01
So making the WebRequest Asynchronous im unsure if that will effect anything — , Apr 08 '16 at 10:02
@ShinyMK: Sure, you are blocking the main application thread with long operations and it can't update window contents. `Application.DoEvents()` helps sometimes, but not always. — Paul, Apr 08 '16 at 10:03
@Paul is there anything I can do then? I want the Proxies to come into the Listbox as they are scraped and I want the Progress Bar to show up While the scraping is happening — , Apr 08 '16 at 10:04
@ShinyMK: You can apply "Just do it" to an example from another question I've linked. — Paul, Apr 08 '16 at 10:05
@Paul so your saying I should try adding the Async? Think that would work? — , Apr 08 '16 at 10:06
You need to look up threading and tasks to understand how to update an UI while application is running. — Allan S. Hansen, Apr 08 '16 at 10:06
@AllanS.Hansen Any chance you could gimme an overview of some kind? :/ — , Apr 08 '16 at 10:07
It is a large area to give an overview off, so when it seems like you're starting from the ground - I'd try and search for some tutorials regarding threading, progress bars, background worker processes etc. Threading had been made a lot easier in later .NET versions, but it's still a large topic. — Allan S. Hansen, Apr 08 '16 at 10:08
@AllanS.Hansen I have a Tutorial Playlist from back in 2008 would that work fine or is it different syntax etc now? Also when you say .net versions do you mean .Net Framework etc? — , Apr 08 '16 at 10:09
I'd find a newer source depending on which version you're using - and yes, for example .NET framework and the VB.NET (and C#) versions. There's been much change. The theory behind is pretty much the same, but .NET helps a lot with Tasks, worker proccesses, parallel namespaces etc. — Allan S. Hansen, Apr 08 '16 at 10:10
@AllanS.Hansen Could you send me a Tutorial/e.tc for .NET Framework 4.6.1? — , Apr 08 '16 at 10:11
Have you ever thought of replacing the many lines by `For i As Integer = 1 To 30 : sources(i + 51) = String.Format("http://nntime.com/proxy-list-{0:00}.htm", i) : Next`? Using a `List(Of String)` instead of an array would make it even easier, because then you don't need to keep track of the size and the index any more. The list just grows automatically as you add items. See: http://www.dotnetperls.com/list-vbnet. — Olivier Jacot-Descombes, Apr 08 '16 at 12:03
@OlivierJacot-Descombes I havent and its a Good idea but then i'd need to code in some un-necessary way of like checking if im using a list and such and such and it will just get in the way of the Foreach for the Array, Honestly its good but its unnecessary. — , Apr 08 '16 at 12:15

Doliveras · Accepted Answer · 2016-04-08T11:10:43.580

0

To allow your UI thread to refresh and show the information while you’re iterating through the proxies you need to use a separate thread to do the job.

You can use the .NET control Backgroundworker to do this. With this control you can start a thread to do the job and this thread can inform the UI thread of the state of the operation using the events of the Brackgroundworker, then the UI thread can update the UI accordingly.

Here you can find some examples of how to use a backgroundworker.

A basic sample of this could be something like:

Imports System.ComponentModel

Public Class Form1

    Private ListOfProcessedElements as List(of integer)


    Private Sub backgroundWorker1_DoWork(ByVal sender As System.Object,ByVal e As DoWorkEventArgs) Handles backgroundWorker1.DoWork
        'Here you do your worker

        'Add an element to the list of Processed elements
        ListOfProcessedElements.add(ANY_NUMBER)


        worker.ReportProgress(i * 10)
    End Sub

    ' This event handler updates the progress.
    Private Sub backgroundWorker1_ProgressChanged(ByVal sender As System.Object, ByVal e As ProgressChangedEventArgs) Handles backgroundWorker1.ProgressChanged
        'Here you can access to the shared variable ListOfProcessedElements and do whatever you want in the UI 
    End Sub

    ' This event handler deals with the results of the background operation.
    Private Sub backgroundWorker1_RunWorkerCompleted(ByVal sender As System.Object, ByVal e As RunWorkerCompletedEventArgs) Handles backgroundWorker1.RunWorkerCompleted

    End Sub

End Class

edited Apr 08 '16 at 11:10

answered Apr 08 '16 at 10:24

Doliveras

1,794
2
14
30

Ok yeah I am following this Tutorial: https://www.youtube.com/watch?v=jv4FdoaUMQE But im getting an issue: https://i.gyazo.com/0270b5539e91cae9c37e3c8c6e1cc8c3.png My Current Code: http://pastebin.com/P6g5YM5J – Apr 08 '16 at 10:33
@ShinyMK You can't "touch" a control created by the UI thread from the background thread. You can put your results on a list or any other variable and then inform the UI thread of the changes using the ReportProgess method, that fires a ProgressChanged event. The UI thread can then update the controls from the function used to capture the event. You can find an example of this on the page I linked in my answer – Doliveras Apr 08 '16 at 10:39
Okay well im trying to use BackgroundWorker1.ReportProgress({arraykey/num}) How do I replace {arraykey/num} with the like Array Number? Like im using "For Each element As String In sources" do I type Element? Im trying to get it to output which "source/element" its doing but not the actual value. Like lets say its doing "sources(97) = "http://txt.proxyspy.net/proxy.txt"" I want it to output 97 but how? – Apr 08 '16 at 10:46
@ShinyMK ReportProgess is not designed to return arbitrary values, is designed to report progess, so it takes values, tipically between 1 and 100, but this is configurable. What you have to do is use a global variable of your own to pass this information between threads – Doliveras Apr 08 '16 at 11:03
Yeah thats exactly why I want it to the the Array KEY so I can make progress tracking much easier on my end. I set the Maximum Value of 100 to 122 which is the last array key. so now if I put ReportProgress inside the foreach and somehow get the array key it would get the progress to work much better and more precise right? But how do I get the Array key – Apr 08 '16 at 11:11
I've added a simple example to the answer – Doliveras Apr 08 '16 at 11:11
@ShinyMK you can change the "For each" for a "for i =0 to XX" and the "i" will be you KEY – Doliveras Apr 08 '16 at 11:14
Worked perfect thanks for that advice! I used the Incrementation advice differently but it worked out! – Apr 08 '16 at 11:34

Visual Basic - Issue with how the code is being Executed

1 Answers1