How to scrape an updating HTML table using Selenium?

Question

I am looking to scrape the coin table from link and create a CSV file datewise. For every new coin update, a new entry at the top should be created in the existing data file.

Desired output

Coin,Pings,...Datetime

BTC,25,...07:17:05 03/18/21

I haven't reached far but below is my attempt at it

from selenium import webdriver
import numpy as np
import pandas as pd

firefox = webdriver.Firefox(executable_path="/usr/local/bin/geckodriver")
firefox.get('https://agile-cliffs-23967.herokuapp.com/binance/')

rows = len(firefox.find_elements_by_xpath("/html/body/div/section[2]/div/div/div/div/table/tr"))
columns = len(firefox.find_elements_by_xpath("/html/body/div/section[2]/div/div/div/div/table/tr[1]/th"))

df = pd.DataFrame(columns=['Coin','Pings','Net Vol BTC','Net Vol per','Recent Total Vol BTC', 'Recent Vol per', 'Recent Net Vol', 'Datetime'])

for r in range(1, rows+1):
    for c in range(1, columns+1): 
        value = firefox.find_element_by_xpath("/html/body/div/section[2]/div/div/div/div/table/tr["+str(r)+"]/th["+str(c)+"]").text
        print(value)
        
#         df.loc[i, ['Coin']] =

score 1 · Accepted Answer · edited Jun 11 '22 at 19:15

Since the data is loaded dynamically you can retrieve it directly from the source, no Selenium needed. It will return json with rows with |-delimited values that need to be split and can be appended to the DataFrame. Since the site updates once per minute, you can wrap everything in a while True that makes the code run every 60 seconds:

import requests
import time
import json
import pandas as pd

headers = ['Coin','Pings','Net Vol BTC','Net Vol %','Recent Total Vol BTC', 'Recent Vol %', 'Recent Net Vol', 'Datetime (UTC)']
df = pd.DataFrame(columns=headers)

s = requests.Session()
starttime = time.time()

while True:
    response = s.get('https://agile-cliffs-23967.herokuapp.com/ok', headers={'Connection': 'keep-alive'})
    d = json.loads(response.text)
    rows = [str(i).split('|') for i in d['resu'][:-1]]
    if rows:
        data = [dict(zip(headers, l)) for l in rows]
        df = df.append(data, ignore_index=True)
        df.to_csv('filename.csv', index=False)
    time.sleep(60.0 - ((time.time() - starttime) % 60.0))

You're welcome. Do note that the `csv` will be overwritten when you rerun the code. If you want to continue working with an existing `csv` file you can load it at the beginning of the code with `df = pd.read_csv('filename.csv')` — RJ Adriaansen, Mar 18 '21 at 10:07

score 0 · Answer 2 · answered Mar 18 '21 at 07:57

0

You can append row data to a DataFrame by putting it into a dictionary:

# We reuse the headers when building dicts below
headers = ['Coin','Pings','Net Vol BTC','Net Vol per','Recent Total Vol BTC', 'Recent Vol per', 'Recent Net Vol', 'Datetime']
df = pd.DataFrame(columns=headers)

for r in range(1, rows+1):
    data = [firefox.find_element_by_xpath("/html/body/div/section[2]/div/div/div/div/table/tr["+str(r)+"]/th["+str(c)+"]").text \
                for c in range(1, columns+1)]
    row_dict = dict(zip(headers, data))
    df = df.append(row_dict, ignore_index=True)

answered Mar 18 '21 at 07:57

joao

2,220
2
11
15

Thank you for a helpful solution. – Neil RT Mar 18 '21 at 10:03
No problem. Since you're new to StackOverflow (SO), please have a look at [this page](https://stackoverflow.com/help/someone-answers) to learn the SO way of doing things. In particular, you might consider accepting @RJ Adriaansen's answer, if that solved your problem. – joao Mar 18 '21 at 10:10

How to scrape an updating HTML table using Selenium?

2 Answers2