Web Scraper - Run Time Error 13 on .document.getElementById("")

Question

First web scraping project!

I've been copying various web scraping code from here but can't get around a

run time error 13: Type Mismatch

on the .document.getElementById("") line I'm using to set a variable for a hyperlink I want to click. I figured it should be treated like the log-in button that I successfully coded. I'm not sure if I am missing a library I should be using, as pretty much every other post had different issues and solutions than what I'm running into. What am I doing wrong here?

I'm using IE11 and Excel 2010. I started adding libraries I thought might provide a solution. The libraries I've activated are as follows:

Visual Basic For Applications
Microsoft Excel 14.0 Object Library
OLE Automation
Microsoft Office 14.0 Object Library
Microsoft HTML Object Library
Microsoft Internet Controls
Microsoft XML, v6.0
Microsoft Shell Controls And Automation

Here is the code and HTML DOM snippet:

Sub IEScrape()
  'we define the essential variables
 Dim ie As Object
 Dim pwd, username
 Dim button
 Dim MemAss

'add the "Microsoft Internet Controls" reference in your VBA Project indirectly
 Set ie = New InternetExplorerMedium
 With ie
     .Visible = True
     .navigate ("internalwebsite.com")
     While ie.readyState <> 4
         DoEvents
     Wend

     Set username = .document.getElementById("userid") 'id of the username control (HTML Control)
     Set pwd = .document.getElementById("password") 'id of the password control (HTML Control)
     Set button = .document.getElementById("loginbtn") 'id of the button control (HTML Control)
     username.Value = "username"
     pwd.Value = "password"
     button.Click
     While ie.readyState <> 4
         DoEvents
     Wend

    'Run time error 13: Type mismatch on next line!!!
    Set MemAss = .document.getElementById("Menu:membershipassociation") 'id of the link (HTML Control)
    MemAss.Click
    While ie.readyState <> 4
         DoEvents
    Wend

 End With



 Set ie = Nothing
 End Sub

Try just appending `.Click` on the same line, without setting variables (`MemAss` in this case). `.document.getElementById("Menu:membershipassociation").Click`. — Nicholas Kemp, Mar 19 '18 at 19:35
@ashleedawg Definitely some of them. I'd say the last two and maybe the OLE Automation I'm not but don't know how to check. I just started adding them willy-nilly to see if they would help. — plankton, Mar 19 '18 at 20:08
I think you're probably not using `Microsoft HTML Object Library, Miicrosoft XML, v6.0, Microsoft Shell Controls And Automation. Also, make sure you have `Option Explicit` at the very top of your module (always). These things won't necessarily help you find *this* bug but it will help prevent others. For this issue, perhaps you could use `getElementsByClassName` and grab the class instead. Are you sure you're at the right section of HTML? (Does the value of `document.innerhtml` look like what you're expecting?) — ashleedawg, Mar 19 '18 at 20:32
also, check out some of the (**many**) other examples on this site. — ashleedawg, Mar 19 '18 at 20:33
Would be nice if you could post the full html code, this way it's impossible to tell what might be wrong. If not, right click the element and post the full css-path/selector to the element you want to access. — drec4s, Mar 19 '18 at 21:18
Have you tried just grabbing the `` tag, check if it is the 1 you want, and then trying to activate it with `FireEvent("onclick")` or just a `.Click`? — Foxfire And Burns And Burns, Mar 20 '18 at 09:42
`Set ie = New InternetExplorerMedium` ???? Just wondering, should not it be `Set ie = New InternetExplorer`? — Foxfire And Burns And Burns, Mar 20 '18 at 09:44
@ashleedawg I removed everything but the default references and the internet controls. I'm not sure exactly how I would implement the code to search by class name then go specifically to menu link that has it's own ID. Initially, I read through the html to find the section but I also used the "Select Element" point and click functionality that validated the section I found. — plankton, Mar 20 '18 at 12:29
@NicholasKemp From the one post I've read on `iFrame` so far, it's certainly a possibility but I'm going to have to look more into how I would implement that. — plankton, Mar 20 '18 at 12:33
@drec4s I would love to do that but it's an internal website so I would definitely get in trouble posting all that code. Is there a cut down version I can post that would still be effective but not give any non-general info? — plankton, Mar 20 '18 at 12:35
@FoxfireAndBurnsAndBurns I'm still very new to working with html so I'll have to look into grabbing the `` tag, but that sounds like it would work. As for `Set ie = New InternetExplorerMedium` I initially had it as the non-Medium version and it would get stuck on `While ie.readyState <> 4` `DoEvents` `Wend` so I switched it and it worked fine until I got to this point. — plankton, Mar 20 '18 at 12:39
@ashleedawg I started playing around with the class name elements and added a `MsgBox` to display any length there may be (no reason). for some reason, after clicking the `OK` button, it accepted `.document.getElementById("Menu:membershipassociation").Click`. Any idea why it works all of a sudden? Do I need to pause the macro? `MsgBox (ie.document.getElementsByClassName("main_menu_link").Length)` `.document.getElementById("Menu:membershipassociation").Click` — plankton, Mar 20 '18 at 13:05

score 2 · Accepted Answer · answered Mar 20 '18 at 14:32

As you have mentioned that pausing the code for 5 seconds allows the code to function I would assume that there is something happening asynchronously to the HTML loading either a AJAX request or JavaScript editing the DOM.

This would mean that once the HTML has loaded (Readystate = 4) the JavaScript could still be running or we could still be waiting for the AJAX response.

Waiting the code as you have would allow internet explorer to finish all of its tasks before VBA can pick up the references. Although the drawback is that you are waiting for an arbitrary amount of time and there is a change it will not load in this interval.

In order to build a more robust control (if needed) I would suggest loading the webpage outside of VBA and using the browsers debugger menu add a breakpoints on any DOM changes, then wait until you can see when your "Menu:membershipassociation" being defined. I would then pay attention to what process called this and see how you can tie your script in. An ideal outcome would be if this data is stored in the page when it loads or in another location you can reach your VBA to directly.

Although when I have hit this roadblock in the past I have used an iterator to have a go at regular intervals which may be able to speed up your code at this section. I also like to use these iterators on any of the DOM that I am not 100% sure is available immediately. essentially just try to load the code every second or 0.5 seconds until it has loaded.

The other suggestion that I would have if when you debug the webpage in a browser, if the data is available when page is loaded then the issue could be due to the fact that you are trying to call the click method straight away. You could try using the IE_DocumentComplete event to signal this is available. An Example has been posted Here which may be of help.

If you are able to update us with what you find from debugging the page as it loads we can point you in a better direction to solve the issue.

score 0 · Answer 2 · answered Mar 20 '18 at 13:28

I have no idea why this works, but I paused the process for 5 seconds and all of a sudden, it recognizes .Document.getElementById("Menu:membershipassociation").Click. If anyone has any critiques on my process, you can post an answer with the better code and I'll mark it correct.

Code Below:

Option Explicit

Sub IEScrape()
  'we define the essential variables
 Dim ie As Object
 Dim pwd, username
 Dim button
 Dim MemAss

'add the "Microsoft Internet Controls" reference in your VBA Project indirectly
 Set ie = New InternetExplorerMedium
 With ie
     .Visible = True
     .Navigate ("internalwebsite.com")
     While ie.ReadyState <> 4
         DoEvents
     Wend

     Set username = .Document.getElementById("userid") 'id of the username control (HTML Control)
     Set pwd = .Document.getElementById("password") 'id of the password control (HTML Control)
     Set button = .Document.getElementById("loginbtn") 'id of the button control (HTML Control)
     username.Value = "username"
     pwd.Value = "password"
     button.Click
     While ie.ReadyState <> 4
         DoEvents
     Wend

    Application.Wait (Now + TimeValue("0:00:05"))
    .Document.getElementById("Menu:membershipassociation").Click

    While ie.ReadyState <> 4
         DoEvents
    Wend

 End With



 Set ie = Nothing
 End Sub

Web Scraper - Run Time Error 13 on .document.getElementById("")

2 Answers2

Linked