HTML Search Code Runs As Intended when Debugging, but not completely when fully Ran

Question

I have a code that runs perfectly when stepping through, but it doesn't run completely when ran as whole. There are no errors. The code searches through HTML tags and scans for a keyword, if that keyword is found within an HTML cell, the whole cell is outputted to an Excel cell. I am looking for a keyword that appears 3 times on a webpage. When I debug and step through, all 3 instances of the keyword are found, and the cell it is in populates cells A1:C1 in Excel. However, when the code is ran as a whole, only the first two instances are found. Can anyone tell me why that is? My guess is the IF statement is not triggered for the last keyword for some reason... I will include the HTML body and my VBA code.

<HTML>
<HEAD>
<META NAME="name" CONTENT="sldfkjsd">
<TITLE>Title</TITLE>

<P>
<TABLE BORDER=3 CELLPADDING=6 CELLSPACING=3>
<TR>
<TD>
<PRE>

AAAA AAAA AAAA AAA AAA AAAA AAA AAAA AAAA AAA
  BBBBb BBBBB BBBBB BBBBB BBBBBBBBBBBBBB B
  keyword
  CCCCCCCcc CCCCCCCCCC CCCCCCCCCCCC DDDDDDDD

</PRE>
</TD>
</TR>
<TR>
<TD>
<PRE>
AAAA AAAA AAAA AAA AAA AAAA AAA AAAA AAAA AAA
  BBBBb BBBBB BBBBB BBBBB BBBBBBBBBBBBBB B
  keyword
  CCCCCCCcc CCCCCCCCCC CCCCCCCCCCCC DDDDDDDD

</PRE>
</TD>
</TR>
<TR>
<TD>
<PRE>
AAAA AAAA AAAA AAA AAA AAAA AAA AAAA AAAA AAA
  BBBBb BBBBB BBBBB BBBBB BBBBBBBBBBBBBB B
  keyword
  CCCCCCCcc CCCCCCCCCC CCCCCCCCCCCC DDDDDDDD

</PRE>
</TD>
</TR>
<NEXT></NEXT>
</TABLE>

Sub subFindScrollIE()

'Set Variables
Dim boolFound As Boolean
Dim strTemp() As Variant, txt As String
Dim strOutput As String
Dim tbl As HTMLTable, tables As IHTMLElementCollection
Dim tr As HTMLTableRow, r As Integer, i As Integer
Dim tRows As IHTMLElementCollection
Dim ie As InternetExplorer
'Delete Rows and Select A1
Rows("1:100").Delete
ActiveSheet.Range("A1").Select


Set ie = New InternetExplorer

'Show Webpage (optional)
'ie.Visible = True
Dim dateToday As String

'Get Today's date in yyyymmmdd to input into URL for most today's alert page
dateToday = Format((Now), "yyyymmdd")


'Navigate to Desired URL with today's date
ie.Navigate "URL" & dateToday & ".html"

'Input Keywords
strTemp = Array("Keyword")

'Wait for IE page to finish loading
Do Until ie.ReadyState = READYSTATE_COMPLETE Or ie.ReadyState = READYSTATE_INTERACTIVE
    'DoEvents
Loop

    'Declare rows
Set allrows = ie.Document.body.getElementsByTagName("tr")

' loop through rows
For Each trow In allrows

    Set tcell = trow.Cells

    ' loop through cells
    For r = 0 To (tcell.Length - 1)

        Set td = tcell(r)

        ' loop through search text
        For i = 0 To UBound(strTemp)
            ' search row for string
            txt = LCase(td.innerHTML)
            If (InStr(txt, LCase(strTemp(i))) > 0) Then

             Application.PrintCommunication = True
                ' search string found. create output html table
                'strOutput = strOutput & tRows(r).outerHTML & vbCrLf
                strOutput = tcell(r).innerHTML

                'Output string to new workbook
                ActiveCell.Value = strOutput

                'Move one cell to the right for next possible match to be pasted
                ActiveCell.Offset(0, 1).Select


                'Reset string (just in case, not needed)
                Debug.Print strOutput
                strOutput = ""
            End If
        Next i
    Next r
Next trow
End Sub

I think some of your troubles could be fixed by [avoiding using `.Select`](http://stackoverflow.com/questions/10714251/how-to-avoid-using-select-in-excel-vba-macros). When it gets to `ActiveCell`, that's *likely* not the cell you're expecting it to be. — BruceWayne, Jun 22 '16 at 17:42
With web scraping, if it works when stepping through but not when run straight (without any pauses) that often points to a timing issue: additional content may be being loaded *after* the page reports as "complete". Try adding a pause before trying to access any content. — Tim Williams, Jun 22 '16 at 17:51
@TimWilliams I added an Application.Wait(Now + TimeValue("0:00:01")) in the first line below the if statement and still the same problem. Do you think it should go somewhere else? — A.Cod, Jun 22 '16 at 19:13
@BruceWayne, I commented out the active Cell and Select lines and left the Debug.Print strOutput, and still only getting the first two keywords in the Immediate window. — A.Cod, Jun 22 '16 at 19:13
Try a 10sec wait directly after the "Wait for IE page to finish loading" loop. — Tim Williams, Jun 22 '16 at 19:22
@TimWilliams, THANK YOU SO MUCH! I moved the application.wait to right below my do until readystate Loop and it works! I never would have guessed it was because the whole page wasn't finished loading i thought thats what the do Until Ready state loop was for! guess it doesn't hurt to be redundant when it comes to vba. — A.Cod, Jun 22 '16 at 19:23
That "wait" loop only gets you so far - in the "old days" of the web it would have been enough, but these days there's often dynamic content which gets loaded only *after* the main page content has completed. — Tim Williams, Jun 22 '16 at 19:26

score 0 · Answer 1 · answered Jun 22 '16 at 19:25

Found the Answer thanks to @TimWilliams.

I put

  Application.Wait (Now + TimeValue("0:00:01"))

Below the

Do Until ie.ReadyState = READYSTATE_COMPLETE Or ie.ReadyState =        READYSTATE_INTERACTIVE
        'DoEvents
    Loop

and now the code runs as intended!

HTML Search Code Runs As Intended when Debugging, but not completely when fully Ran

1 Answers1