Real time websites scraping with selenium and python

Question

I am doing trading in a real time website. Now I want to scrap it continuously to extract real time stock or currency data for calculations using selenium and python. How do I approach the same. The webpage data is getting updated continuously and shown in the attached image in highlighted colours. Every time the data changes I should be able to extract it and do some calculations in the code. Pls help me to achieve the same.

No sir, I am totally clueless here. I mean I can easily scrap static websites, but if I scrap this webpage continuously in loop, my IP will get blocked. What I mean if there is a way to open a session and feed data continuously whenever any change is detected in the webpage elements. — Soumya Chakraborty, Apr 06 '22 at 06:20
You are asking a whole new question in comment section. Is is possible to share the URL? — cruisepandey, Apr 06 '22 at 06:22
Sir , I actually don't know how to approach the problem so mentioned those. I would highly appreciate if you can help me with any method. The url is : https://www.canmoneyonline.com/login.aspx. But it will require login id and password. A close example is : https://in.investing.com/ . — Soumya Chakraborty, Apr 06 '22 at 06:32
On this website https://in.investing.com/ you want to scrape 6M nifty 50, BSE Sensex value? — cruisepandey, Apr 06 '22 at 06:59
Sir in this page I want to scrap 1D Nifty 50, 17,892.30 full Xpath - /html/body/div/div[4]/aside/div/section[1]/section[1]/div[1]/section/div/div/table/tbody/tr[1]/td[2]/span — Soumya Chakraborty, Apr 06 '22 at 07:11

cruisepandey · Accepted Answer · 2022-04-06T07:52:47.877

1

to capture the dynamically changing value of Nifty50 for the 1D segment.

You should use the below XPath:

//button[text()='1D']/ancestor::nav//following-sibling::section/descendant::a[text()='Nifty 50']/../following-sibling::td/span

Your effective code would be:

driver.maximize_window()
wait = WebDriverWait(driver, 20)

driver.get('https://in.investing.com/')

i = 0
while True:
   time.sleep(5)
   oneD_Nifty50 = wait.until(EC.visibility_of_element_located((By.XPATH, "//button[text()='1D']/ancestor::nav//following-sibling::section/descendant::a[text()='Nifty 50']/../following-sibling::td/span")))
   print(oneD_Nifty50.text)
   i = i + 1

   if i == 10:
       break
   else:
       continue

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output:

17,828.80
17,828.65
17,828.40
17,828.70
17,828.70
17,827.20
17,827.70
17,827.70
17,828.80
17,828.80

Process finished with exit code 0

edited Apr 06 '22 at 07:52

answered Apr 06 '22 at 07:39

cruisepandey

28,520
6
20
38

Thanks sir, but how to continuously fetch data and print. – Soumya Chakraborty Apr 06 '22 at 07:47
Updated above, you can choose any value for i where you want to come out from the loop. In this example I have just given `if i == 10:` you can try with 50 or 100 or any other number infact also I am retrieving data after 5 secs. – cruisepandey Apr 06 '22 at 07:54
1

Thanks sir for the help, I though the website will block the IP due to continuous pinging but it isn't. I will try the same in my trading website. – Soumya Chakraborty Apr 06 '22 at 08:01

Real time websites scraping with selenium and python

1 Answers1