0

I have written a small python code to scrape a table in a web page. It uses qt4 to scrape. Now, The problem is I need to keep scraping the data every 5 mins. I am thinking of refreshing the page and scrape again. How can I refresh the webpage and scrape again every 5 mins? Below is the code which I am using to scrape. 

import sys  
from BeautifulSoup import BeautifulSoup
from PyQt4.QtGui import *  
from PyQt4.QtCore import *  
from PyQt4.QtWebKit import *  
from lxml import html 
import redis
from time import sleep

class Scraper(QWebPage):  
    def __init__(self, url):  
        self.app = QApplication(sys.argv)  
        QWebPage.__init__(self)  
        self.loadFinished.connect(self._loadFinished)  
        self.mainFrame().load(QUrl(url))  
        self.app.exec_()  
        #self.render = Scraper(url)

    def _loadFinished(self, result):  
        self.frame = self.mainFrame()  
        self.app.quit() 
    def close_app(self):
        self.app.quit()
        print "closed"

url = 'https://www.nseindia.com/live_market/dynaContent/live_analysis/top_gainers_losers.htm?cat=G' 

r = Scraper(url)  
result = r.frame.toHtml()
formatted_result = str(result.toAscii())
soup = BeautifulSoup(formatted_result)
table = soup.find(id="topGainers")

print table
Chiyaan Suraj
  • 1,021
  • 2
  • 13
  • 27

2 Answers2

0

Check this page out. It provides a very light-weight library for scheduling tasks and should work fine within Qt. How do I get a Cron like scheduler in Python?

But if you are worried about your GUI freezing or just wanting to keep everything native within Qt, check this out : Background thread with QThread in PyQt.

Community
  • 1
  • 1
Mike JS Choi
  • 1,081
  • 9
  • 19
0

You can use QtCore.QTimer.singleShot(5 * 60, func) function.

def __init__(self, url):
   # ... 
   self.show_page()

def show_page(self)
   # display page here 

   QtCore.QTimer.singleShot(5 * 60, self.show_page)
alexey
  • 706
  • 5
  • 9