0

I am scraping a web page with PyQt. In order to facilitate this, I have constructed the following class:

class Client (QWebEnginePage):
def __init__(self, url):
    self.app = QApplication(sys.argv)
    QWebEnginePage.__init__(self)
    self.html = ''
    self.loadFinished.connect(self._on_load_finished)
    self.load(QUrl(url))
    self.app.exec_()

def _on_load_finished(self):
    self.html = self.toHtml(self.Callable)
    print('Load finished')

def Callable(self, html_str):
    self.html = html_str
    self.app.quit()

I then use this class to obtain data from multiple subpages that I loop over with a for loop:

from config import o_u_types,countries,leagues,limited_league_countries,bookmakers

def getodds(url):

    for i in o_u_types:

        ou_i_df= pd.DataFrame()

        url_appendix="#over-under;2;{:.2f};0".format(i)
        o_u_type= i
        o_u_match_url= str(url)+str(url_appendix)
        print(o_u_match_url)
        ou_page=Client(o_u_match_url)

#Here some actions are performed, after which the output is store in a data frame (data_df) that is then appended to the master_df that stores the data across loops

       pd.concat([master_df,data_df])
    print(master_df)
    return master_df

After opening the client with the individual match URL, i perform some further actions that should not be relevant to the issue (but happy to include them if needed)

This works fine for the first iteration, but execution always concludes without an error message before moving on to the second execution.

What might be the issue here?

0 Answers0