1

To prepare for the eventual 'going away' of IE11, I've been trying to figure out how to replace a couple parts of my code. One involves launching IE and using that browser to scrape some pages. Is there an equivalent way to do the below in Edge? I don't see a way to add a reference to the Edge libraries like I did with 'Microsoft Internet Objects' and IE11.

    Dim ie As InternetExplorerMedium: Set ie = New InternetExplorerMedium
    Dim html As HTMLDocument

    With ie
        .Visible = False
        .Navigate website 'string that's created above this code
    End With

    Do While ie.ReadyState <> READYSTATE_COMPLETE
        DoEvents
    Loop
    
    Application.Wait Now + #12:00:10 AM#
    
    Set html = ie.Document

Thanks everyone for your help.

Sudio
  • 153
  • 1
  • 9
  • Microsoft stop updating VBA few years ago, unless public create Edge library, then VBA can only use IE for web scraping.. – Kin Siang May 21 '21 at 11:53
  • 2
    You can use SeleniumBasic to scrape webpages with other browsers than the IE with VBA. Look at the first answer here: https://stackoverflow.com/questions/57216623/using-google-chrome-in-selenium-vba-installation-steps/57224810 – Zwenn May 21 '21 at 12:02
  • That's a good suggestion. I don't have admin rights on my machine however, and no Chrome browser (just IE and Edge). – Sudio May 21 '21 at 14:39

1 Answers1

1

Ok, a few explanations. I am writing these as a reply so as not to have to split them into several comments.

Does Edge work instead of IE to do web scraping with VBA?

It does not work directly. The reason is that IE has a COM interface (Wikipedia: Component Object Model). No other browser has this interface. Not even Edge.

But for Edge there is also a web driver for Selenium. Even provided directly by MS.

Another alternative - xhr

Since you can't use Selenium because you don't have admin rights, there might be the possibility to use xhr (XML HTTP Request). However, in order to make a statement on this, we would have to know the page that you want to scrape.

Xhr can be used directly from VBA because it does not use a browser. The big limitation is that only static content can be processed. No JavaScript is executed, so nothing is reloaded or generated dynamically in any other way. On the other hand, this option is much faster than browser solutions. Often, a static file provided by the web server is sufficient. This can be an HTML file, a JSON or another data exchange format.

There are many examples of using xhr with VBA here on SO. Take note of the possibility first as another approach. I can't explain the method exhaustively here, also because I don't know everything about it myself. But there are many ways to use it.

By the way

IE will finally be discontinued in June 2022 and will then also no longer be delivered with Windows. That's what I read on the German IT pages a few days ago. But there are already massive restrictions on the use of IE.

Zwenn
  • 2,147
  • 2
  • 8
  • 14
  • Thank you so much for the detailed explanation. The site I'm using (not publicly accessible) uses JavaScript/React, so I don't believe that xhr will work (I may be wrong). I'll check out Selenium option though! – Sudio May 22 '21 at 02:34