IE in vba code lose connection

Question

I made a macro that scrap in formation from the website www.soccer24.com I want to get all the details of every match in the home page.

I already made the code and it works but when the macro is running, suddenly, the IE conection is lost. I mean, the macro have to go into 150 different websites inside the same page, like this:

https://www.soccer24.com/match/Qo8j57nO/#odds-comparison;1x2-odds;1st-half

the only thing that changes in each link is the id that is bolded

the macro goes to the website, pulls some information, and goes to another, pulls information, and so on.

when the macro is about the page 40, the IE connection just breaks, and it doesn't work anymore and I can't go to any website. When I close the program and open it again, I recover the internet connection, but it's always the same when it goes on the 40 or 50 iteration, the IE connection breaks and doesn't open any website.

its like IE VBA has a "working time limit" o "websites visit limit" I don't know whats happening.

The code That I am using is this one:

Piece of code 1

Piece of code 2

so it works perfectly, but eventually in the 40 or 50 iteration of 150. when it tries to go to a website:

IE.Navigate "https://www.soccer24.com/match/" & Range("A" & i).Value & "/#odds-comparison;1x2-odds;1st-half"

the internet explorer shows "no conection" and it doesn't work anymore. it's like IE VBA has a "working time limit" o "websites visit limit"

Answer 1

In webscraping this should be your best friend:

Option Explicit

Public Sub SaveResponse(response As String) 'don't save as txt, but as html to easily see the structure

Dim fso As Object, oFile As Object
Dim filedir As String

filedir = CreateObject("WScript.Shell").specialfolders("Desktop")

Set fso = CreateObject("Scripting.FileSystemObject")
Set oFile = fso.CreateTextFile(filedir & "\response.html", Unicode:=True) 'Default is ANSI encoding, change to UNICODE

oFile.WriteLine response
oFile.Close

End Sub

For IE automation it is used like this:

SaveResponse myIE.document.body.innerHTML

It will save response as HTML file on your desktop. You will be able to see how last received response before error looked like.

There are multiple things which can go wrong with IE automation, however the last thing I would expect is entirely losing internet connection. This needs to be more closely checked. Will Chrome open this or any other website at the same time IE can't?

To overcome some IE issues, you may want to IE.Quit every X number of requests. Additionally removing old IE instances in background is sometimes helpful, especially to handle memory leak:

Public Sub TerminateIE()

Dim objWMIService As Object
Dim colItems, objItem

Set objWMIService = GetObject("winmgmts:\\.\root\cimv2")
Set colItems = objWMIService.ExecQuery("Select * From Win32_Process")

On Error GoTo ErrHandler
For Each objItem In colItems
    If objItem.Name = "iexplore.exe" Then objItem.Terminate
Next objItem

Exit Sub
ErrHandler:
MsgBox ("error " & Err.Name & "with description " & Err.Description & String(2, vbLf) & "Nothing object test for objItem returns: " & objItem Is Nothing)
End Sub