1

My first msg in stackoverflow... :-) My project: I would like to retrieve data on my internet speed given by the site http://www.bredbandskollen.se/ using a daily python routine, using beautifullsoup.

While I have identified different ways to retrieve the test result I am interested in using bs, I am stuck by the simple fact I do not succeed to launch/activate the speed test (basically what would be a button click).

I could notice there is not a conventional HTML5 form sent. Instead, I saw a "ws" request which I have no idea of what it is.

Does any one has an idea how to initiate the speedtest, simulate the button click or advice me how to treat a ws request?

thx.

Albert Norberg
  • 11
  • 1

1 Answers1

1

Selenium would be a good way to go to simulate the process of opening the page, then clicking the button. Then it waits for the test to finish and grabs the results.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec

driver = webdriver.Chrome("C:/chromedriver_win32/chromedriver.exe")

driver.get("http://www.bredbandskollen.se/")
driver.find_element_by_id("mainStartTest").click()

# Waits until test is complete. Timesout after 60 seconds
WebDriverWait(driver, 60).until(ec.presence_of_element_located((By.XPATH, './/span[@class = "bbk-test-info-value" and text() != ""]')))


# Get the results
results = driver.find_elements_by_xpath('.//span[@class = "bbk-test-box-result"]')
dlSpeed = results[0].text
ulSpeed = results[1].text
ltSpeed = results[2].text

print ('Results\nDownloading: %s\nUploading: %s\nLatency: %s' %(dlSpeed, ulSpeed, ltSpeed))

driver.close()

Output:

Results
Downloading: 39,86 Mbit/s
Uploading: 4,16 Mbit/s
Latency: 240,15 ms

You could still use BeautifulSoup, but that wouldn't come until after the test is run. But not really needed, but at least lets you see how selenium and beautifulsoup are used to find the tags with the data you want:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
import bs4

driver = webdriver.Chrome("C:/chromedriver_win32/chromedriver.exe")

driver.get("http://www.bredbandskollen.se/")
driver.find_element_by_id("mainStartTest").click()

# Waits until test is complete. Timesout after 60 seconds
WebDriverWait(driver, 60).until(ec.presence_of_element_located((By.XPATH, './/span[@class = "bbk-test-info-value" and text() != ""]')))


# Get the results
soup = bs4.BeautifulSoup(driver.page_source, 'html.parser')
results = soup.find_all('span', {'class':'bbk-test-box-result'})

dlSpeed = results[0].text
ulSpeed = results[1].text
ltSpeed = results[2].text

print ('Results\nDownloading: %s\nUploading: %s\nLatency: %s' %(dlSpeed, ulSpeed, ltSpeed))

driver.close()
chitown88
  • 27,527
  • 4
  • 30
  • 59
  • Oj! I didn't expect you give me back all the code! :-) Still trying to finish my own version.... but won't be as lean as yours ! Thx ! – Albert Norberg Jan 14 '19 at 14:08
  • ya give it a shot on your own first, then compare and contrast. Best way to learn! – chitown88 Jan 14 '19 at 14:11
  • Feedback: - I use selenium and it did work well; However I used a basic time.sleep(60); I think your line of code using WebDriverWait is really elegant ! - I also learned with bs4. I didn't catched that all the values were already in one higher level class so I did it for each of them... You code makes it more simple :-) (and yes this confirm I am a beginner ;-)). Thx for the support ! – Albert Norberg Jan 15 '19 at 07:39
  • great! I actually just learned to use WebDriverWait for this question (I originally had just a set 45 seconds time.sleep like you were saying...but I knew of the driver wait, just never implemented it. There is also a way to pull the data with selenium and not need bs4 at all. But I'm more familiar with bs4 – chitown88 Jan 15 '19 at 10:32