0

I'm attempting to extract data from the following site using VBA, by inputting a city, and having selected results outputted into excel cells. I'm very new to this, and this my third attempt, but now I'm getting a "Object Required" error when I try to run it. I've stepped through it, and it throws the error at, of course, the IE object I tried to create. Any suggestions on what I can do to tweak my code? Any help would be much appreciated! Thank you.

Code

Private Sub CreditUnion()

If Target.Row = Range("City").Row And Target.Column = Range("City").Column Then

    Dim IE As Object

    Set IE = CreateObject("internetexplorer.application")


    IE.Navigate "http://mapping.ncua.gov/SingleResult.aspx"
    IE.Visible = False

    Do While IE.Busy

        DoEvents

    Loop

    Set TableResults = IE.document.getElementsByID("MainContent_newDetails")

    Dim City As String: City = TableResults.Cells(17).innerHTML
    Dim CreditUnion As String: CreditUnion = TableResults.Cells(0).innerHTML
    Dim Region As String: Region = TableResults.Cells(9).innerHTML
    Dim Status As String: Status = TableResults.Cells(3).innerHTML
    Dim Assets As String: Assets = TableResults.Cells(13).innerHTML
    Dim Members As String: Members = TableResults.Cells(15).innerHTML


    Range("B1").Value = City
    Range("C4").Value = CreditUnion
    Range("D4").Value = Region
    Range("E4").Value = Status
    Range("F4").Value = Assets
    Range("G4").Value = Members


    IE.Quit
    Set IE = Nothing

End If

End Sub

Code can't get past this point [Code stuck here][1]

We're getting close! made it past the first screen. It's just not pulling in the data now in the case statements [enter image description here][2]

K.K.
  • 49
  • 8

1 Answers1

0

I take New York for example, the code as below.

I rewrite on 2016/6/7

Public Declare Sub Sleep Lib "kernel32.dll" (ByVal dwMilliseconds As Long)

Sub CreditUnion()
    Dim IE As Object, TableResults As Object, webRow As Object, charterInfo As Variant, page As Long, pageTotal As Long, r As Long
    Dim beginTime As Date, i As Long

    Set IE = CreateObject("internetexplorer.application")
    IE.navigate "http://mapping.ncua.gov/ResearchCreditUnion.aspx"
    IE.Visible = True

    Do While IE.Busy Or IE.readystate <> 4   '4 = READYSTATE_COMPLETE 
        DoEvents
    Loop

    'input city name into form
    IE.document.getelementbyid("MainContent_txtCity").Value = "new york"
    'click find button
    IE.document.getelementbyid("MainContent_btnFind").Click
    sleep 5 * 1000

    'total pages
    pageTotal = IE.document.getelementbyid("MainContent_pager_total").innertext
    page = 0

    Do Until page = pageTotal
        DoEvents
        page = IE.document.getelementbyid("MainContent_pager_to").innertext
        With IE.document.getelementbyid("MainContent_grid")
            For r = 1 To .Rows.Length - 1
                If Not IsArray(charterInfo) Then
                    ReDim charterInfo(7, 0) As Variant
                Else
                    ReDim Preserve charterInfo(7, UBound(charterInfo, 2) + 1) As Variant
                End If

                charterInfo(0, UBound(charterInfo, 2)) = .Rows(r).Cells(0).innertext
            Next r
        End With

        If page < pageTotal Then
            IE.document.getelementbyid("MainContent_pageNext").Click
            beginTime = Now
            Application.Wait (Now + TimeValue("00:00:05"))
        End If
    Loop

    For r = 0 To UBound(charterInfo, 2)
        IE.navigate "http://mapping.ncua.gov/SingleResult.aspx?ID=" & charterInfo(0, r)
        Do While IE.Busy Or IE.readystate <> 4   '4 = READYSTATE_COMPLETE 
            DoEvents
        Loop
        'wait 5 sec. for screen refresh
        sleep 5 * 1000

        With IE.document.getelementbyid("MainContent_newDetails")
            For i = 0 To .Rows.Length - 1
                DoEvents
                Select Case .Rows(i).Cells(0).innertext
                Case "Credit Union Name:"
                    charterInfo(1, r) = .Rows(i).Cells(1).innertext
                Case "Region:"
                    charterInfo(2, r) = .Rows(i).Cells(1).innertext
                Case "Credit Union Status:"
                    charterInfo(3, r) = .Rows(i).Cells(1).innertext
                Case "Assets:"
                    charterInfo(4, r) = Replace(Replace(.Rows(i).Cells(1).innertext, ",", ""), "$", "")
                Case "Number of Members:"
                    charterInfo(5, r) = Replace(.Rows(i).Cells(1).innertext, ",", "")
                Case "Address:"
                    charterInfo(6, r) = .Rows(i).Cells(1).innertext
                Case "Phone:"
                    charterInfo(7, r) = "'" & .Rows(i).Cells(1).innertext
                End Select
            Next i
        End With
    Next r


    IE.Quit
    Set IE = Nothing

    'post result on Excel cell
    Worksheets(1).Range("A1").Resize(UBound(charterInfo, 2) + 1, UBound(charterInfo, 1) + 1).Value = Application.Transpose(charterInfo)
End Sub
PaichengWu
  • 2,649
  • 1
  • 14
  • 28
  • Thanks so much for this! I will alter it to make it fit my requirements. I will go through it and hopefully it'll make sense to me with what happens when we get to the ID screen. Again, thanks a lot! – K.K. Jun 04 '16 at 17:19
  • You are welcome. Could you please give me a upvote? – PaichengWu Jun 04 '16 at 22:43
  • @K.K. or accept my answer. So, I can earn reputation. – PaichengWu Jun 04 '16 at 23:43
  • Thanks a lot. I went ahead and accepted your answer and upvoted it. One more question: I get an Object Error when running the code. I step through it, and it won't go past the first `End With` statement, which should be pushing it to the next page with all info I want. Is there something I need to lookout for? Also, can I assign for the City to be a cell in excel so I input in a cell rather than the code? that was my plan @pcw – K.K. Jun 05 '16 at 21:55
  • I renewed my code which add `dim i as long`. I test the code. It works. – PaichengWu Jun 06 '16 at 00:24
  • @K.K. Thank you for accepting my answer. I do not understand your problem about the first `End With` statement. Would you please explain it more clearer. If you want to assign City to a cell, for example A1 in Sheet2, change `IE.document.getelementbyid("MainContent_txtCity").Value = "new york"` to `IE.document.getelementbyid("MainContent_txtCity").Value = worksheets(2).Range("A1").Value`. – PaichengWu Jun 06 '16 at 00:38
  • Hi! I still get an object error for some reason. Everything looks defined. And earlier, I meant to say the code won't output the credit union name and the rest of the information we want in excel. When I step into the code, I can't get past the `Do` loop @pcw – K.K. Jun 06 '16 at 01:29
  • `TableResults`, that I don't see used anywhere. You think that's why the Object error keeps being called? @pcw – K.K. Jun 06 '16 at 01:36
  • I see get to the page where it lists all the credit unions in the city I want, but right after I get to that page the Object Error pops. It seems like it can't get past it. Nothing outputs into excel. It gets stuck at this site: http://mapping.ncua.gov/ResearchCreditUnion.aspx @pcw – K.K. Jun 06 '16 at 01:43
  • It never reaches here: `page = IE.document.getelementbyid("MainContent_pager_to").innertext If page < IE.document.getelementbyid("MainContent_pager_total").innertext Then IE.document.getelementbyid("MainContent_pageNext").Click Loop Until page = IE.document.getelementbyid("MainContent_pager_total").innertext` @pcw – K.K. Jun 06 '16 at 02:11
  • If you run the code step by step (press F8 in VBE), where does it stop? If possible, show the picture of VBE and IE. – PaichengWu Jun 06 '16 at 04:29
  • it never reaches here: `page = IE.document.getelementbyid("MainContent_pager_to").innertext If page < IE.document.getelementbyid("MainContent_pager_total").innertext Then IE.document.getelementbyid("MainContent_pageNext").Click Loop Until page = IE.document.getelementbyid("MainContent_pager_total").innertext` @pcw I will try to post pics now. Thanks a lot! – K.K. Jun 06 '16 at 04:33
  • I just uploaded the picture in my original question. @pcw – K.K. Jun 06 '16 at 04:42
  • I guess it may need more time to refresh the web. Please try to change `Application.Wait (Now + TimeValue("00:00:05"))` to `Application.Wait (Now + TimeValue("00:00:10"))`. – PaichengWu Jun 06 '16 at 04:45
  • No, that doesn't seem to be the fix. Same object error. It doesn't go to the next page at all. It's missing an object. Did this work for you? Did it output the results we wanted? @pcw – K.K. Jun 06 '16 at 04:47
  • I even did 50 seconds. Maybe new york has too many results? I tried Alaska, and that only has two results. It still didn't work. – K.K. Jun 06 '16 at 04:50
  • No! haha. Can you show me what your excel file looks like? When the data is outputted? I would really appreciate it. – K.K. Jun 06 '16 at 04:54
  • I uploaded another picture with alaska and time to 10 seconds. you can see the object error. What can be causing this you think? @pcw – K.K. Jun 06 '16 at 05:01
  • PCW!!!! It made it past the the first loop and onto the second page! But now it's not pulling the data. That's the only thing! I attached a new image. It just keeps going through those cases and repeating itself. @pcw – K.K. Jun 06 '16 at 05:09
  • It skips over the the following. Maybe the rows are incorrectly specified: `Case "Region:" charterInfo(2, r) = .Rows(i).Cells(1).innertext Case "Credit Union Status:" charterInfo(3, r) = .Rows(i).Cells(1).innertext Case "Assets:" charterInfo(4, r) = Replace(Replace(.Rows(i).Cells(1).innertext, ",", ""), "$", "") Case "Number of Members:" charterInfo(5, r) = Replace(.Rows(i).Cells(1).innertext, ",", "")` We're so close! @pcw – K.K. Jun 06 '16 at 05:14
  • There are more than one `Application.Wait beginTime + TimeValue("0:00:05")`. Please change them to `Application.Wait beginTime + TimeValue("0:00:10")`. Or if you want, make it 50. – PaichengWu Jun 06 '16 at 05:16
  • Okay, so here's the deal. In debugging, I just held F8 for it do go through the loops. When it did that, it outputted the results! But, this only works in debug mode. What do you recommend? – K.K. Jun 06 '16 at 05:19
  • My output: [https://photos.google.com/share/AF1QipNI8I9PVlMi6oGOiJpg_kQScPPijDUaJQ4xt19WfHr_ll2BNUHEOnKPm363wbU9rg?key=Zl91d013bXlhc2RLc2tOdGxTdUxBX1NxNzREMkdn](https://photos.google.com/share/AF1QipNI8I9PVlMi6oGOiJpg_kQScPPijDUaJQ4xt19WfHr_ll2BNUHEOnKPm363wbU9rg?key=Zl91d013bXlhc2RLc2tOdGxTdUxBX1NxNzREMkdn) – PaichengWu Jun 06 '16 at 05:30
  • Okay, that's what I'm waiting on now. Thanks a lot man. You've been great help. Talk to you after meeting/tomorrow. I have to sleep now. @pcw – K.K. Jun 06 '16 at 05:38
  • Hi @pcw I tried 5 minutes for the city alaska, and it still through the Object Error. When I'm in debug mode, and keep going through the cases and loops myself it works, but it takes forever. Any idea on what's going on? Thank you. – K.K. Jun 06 '16 at 16:00
  • Hi @pcw, any chance you figured out how to make this run? Someone suggested using the sleeper API, but that hasn't been helpful yet. I wonder why it works for you and not me. I'm using excel 2013. Thank you very much! – K.K. Jun 06 '16 at 21:06
  • You are a genius! Thank you so much man! this thing was driving me crazy all day. You are the best! @pcw – K.K. Jun 07 '16 at 02:36
  • Hi @pcw! I reran it again, after it worked perfectly. But now I get the object error again. Why does this keep happening :( – K.K. Jun 07 '16 at 03:12
  • Website condition and bandwidth could cause error. Try give IE more time to refresh, that is wait 10 or more sec. Besides, the webmaster could ban you, if he/she notes you are keeping scrap the website in a short time. – PaichengWu Jun 07 '16 at 03:31
  • My IE equip the Adblock add-in. – PaichengWu Jun 07 '16 at 03:44
  • hi @pcw, I honestly don't know why the error occurs now. The error is not thrown in debug mode at all. It works fine in debug. I did it for new york earlier and it pulled all the data. you want me to equip my IE with adblock? I think it has it – K.K. Jun 07 '16 at 03:48
  • I added adblock to IE. I tried another city with 76 records and it works @pcw. Anyway I can link that "Find" button in my excel to the whole execution we have? I have attached a new pic in my original question for you to see :) Also, I want to pull the "Address" and "Phone" as well. Is this code correct: `Case "Phone:" charterInfo(6, r) = Replace(.Rows(i).Cells(1).innertext, ",", "") Case "Address:" charterInfo(7, r) = Replace(.Rows(i).Cells(1).innertext, ",", "")` – K.K. Jun 07 '16 at 04:07
  • I tried linking the button to the `Sub CreditUnion` but it required an object to be created @pcw – K.K. Jun 07 '16 at 04:27
  • I linked the button to the sub, but I think the IE just doesn't run consistently when this code is called @pcw ha – K.K. Jun 07 '16 at 04:39
  • I tried debugging and it doesn't throw an error when I debug. Only when I execute it. Someone mentioned that maybe we should not use `IE.Busy` ans use the sleepkernell API. You think that would help @pcw? Because this really feels like it's a timing issue. – K.K. Jun 07 '16 at 05:01
  • I rewrite my code which add `sleep` API, `IE.readystate` and scrap the address and phone. – PaichengWu Jun 07 '16 at 05:31
  • Thank you so much @pcw. You are literally the most wonderful help. I linked my button method and it works good so far. I will keep you updated if anything happens. Thanks again. – K.K. Jun 07 '16 at 18:27
  • :) Thanks a lot @pcw – K.K. Jun 08 '16 at 02:36
  • Hi @pcw, I wanted to know if I can ask you questions about the code you modified, as I want to learn to build my own web scrapping excel. Would that be okay? I would love to learn from you. – K.K. Jun 08 '16 at 19:50
  • Say your questions. I will try my best. – PaichengWu Jun 08 '16 at 23:49
  • Hey @pcw, I will upload my questions as part of the original question I posted, in case other newbies need help. Or, if you'd like, I can send you an email with my questions! Thanks a lot in advance. – K.K. Jun 11 '16 at 18:36
  • Hi @pcw! I posted a new question here, similar to what we have been doing: http://stackoverflow.com/questions/37782725/vba-scrapping-from-website I thought maybe you can help with it pretty easily, since you already know what I'm looking for. I had a question about `pageTotal = IE.document.getelementbyid("MainContent_pager_total").innertext page = 0` - how would I have adapted this to my new macro that I want to create? How would I know where the page sums in the HTML code? Also, why do you minus 1 here: `For r = 1 To .Rows.Length - 1`? And you created `r` so you can sequence loops? – K.K. Jun 13 '16 at 06:02
  • Thanks a lot in advance - @pcw – K.K. Jun 13 '16 at 06:05
  • I saw your new question. What does your total page problem mean (credit union or bank)? – PaichengWu Jun 13 '16 at 06:45
  • Just what would I have to look for in order to total the page? @pcw – K.K. Jun 13 '16 at 15:04
  • Hi @pcw, I made a new question with an updated code that I tried to mimic using what you wrote for the last search: http://stackoverflow.com/questions/37797346/xhtml-website-scraping-guidance Any help to get this new search to work and teaching would be great :) Thank you so much. – K.K. Jun 13 '16 at 19:07
  • hey @pcw, hope all is well. Wanted to see what your thoughts were on the new project. – K.K. Jun 14 '16 at 03:00