Scrape NetGear Switch GS752TPP Information with Python Selenium

Question

I am using the NetGear Switch model GS752TPP for a project. This model is able to supply power-over-ethernet and I would like to automatically scrape that information via the webinterface, because no API exists for that. I need the power consumption of the attached devices over time. See the following picutre to get a better understanding: NetGear Switch Power Monitoring Table

The pynetgear Python script does not work for my model.

I am using Python Selenium to open a headless Firefox to access the webinterface. I can send my password to the respective field, log in and click my way through to the right tab (s. code below). I can see the data in a table, however it is not visible in the HTML. I could not access the fields via CSS or xpath. To find the right CSS or xpath I used the Selenium IDE plugin for Firefox.

After inspecting the websites network I were able to find the right request, which returns the data I need:

URL
https://<ip_address>/cgi/get.cgi?cmd=poe_port&dummy=1612446184975&bj4=07e0349a9b364bf17177eafe167deaa6

cURL
curl 'https://<ip_address>/cgi/get.cgi?cmd=poe_port&dummy=1612446184975&bj4=07e0349a9b364bf17177eafe167deaa6' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'X-CSRF-XSID: xZTyso9GzNQ5sMRarzmkCSJQAssA1WLUlk0Q5cwdPStiUJb0KlE+92EkFgVgroCVlMuOjcR8Rk6EIYCixMl53z+dCunTYwWs0Z76er0EvZPiSGIjCUtYi3BV0VS0OLq6sA32EIPfSDBE/xE5xa/3Uzovxo6Sc8OodurgbgxWGoE=' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'Referer: https://<ip_address>/html/sys_poe_port.html?aj4=2a08be6&bj4=9fa58c963a822462157cbfc45f82c702' -H 'Cookie: testcookie; cookie_language=defLang'

NetGear uses a X-CSRF token to be more secure against attacks. This describes roughly what that is. So I assume I need the following to recreate the URL:

1. Value after cmd=poe_port&dummy= (this one chances for every new session)
2. X-CRSF-XSID token (this one is unfortunately not available in the cookies)

Do I have to run any Javascript by myself vis Selenium or how can I automatically download the power consumption?
Any help is highly appreciated and I will provide you with any missing information.

Thanks and regards,
René

Code to access NetGear webinterface till data export.

from selenium import webdriver
from selenium.webdriver.common.by import By
profile = webdriver.FirefoxProfile()
profile.accept_untrusted_certs = True
options = webdriver.FirefoxOptions()
options.headless = True
driver = webdriver.Firefox(executable_path="/home/ubuntu/geckodriver", firefox_options=options, firefox_profile=profile)
driver.get("https://<ip_address>")
driver.find_element(By.ID, "password").send_keys(<password>)
driver.find_element(By.ID, "local_login").click()
driver.find_element(By.ID, "menu_fld2SysPoE").click()
driver.find_element(By.ID, "menu_fldAdv").click()
driver.find_element(By.ID, "menu_doc4SysPoEPort").click()

UPDATE I

After installing npm on the Raspberry Pi, I installed the gs310tp package from Taisuke Yamada. You find the package here. And then ran it with .node_modules/gs310tp/bin/gs310tp.js -u https://<ip_address< -p <password> poe status. However, this gives the error (node:444693) UnhandledPromiseRejectionWarning: Error: self signed certificate. To disable this message one can run export NODE_TLS_REJECT_UNAUTHORIZED='0' before. But note that this decreases the security.

A simple Python script which incorporates the Javascript looks like the following

import subprocess
import re

p = subprocess.Popen(["./gs310tp.js", "-u https://ip", "-p SecurePassword", "poe", "status"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.wait()

out, err = p.communicate()
out = bytes.decode(out)  # Convert from byte to string
# Conversion from string to dict is quite tricky and I dont need that so I used regex instead

out = out.replace('"', '')
out = out.replace("'", "")
voltage = re.findall("voltage: \d+", out)
ampere = re.findall("amphere: \d+", out)
power = re.findall("power: \d+", out)

Answer 1

I recently did the same automation for NetGear GS310TP PoE switch. It uses pair of "get.cgi" and "set.cgi" for management just like your model, so both models might share the same method for protection.

Obtaining key for X-CSRF-XSID: is a bit involved process.

You first have to obtain one-time temporary key using the API (cmd=home_loginStatus API), few seconds after you have submitted correct password (to cmd=home_loginAuth API). This key can only be fetched once, and will invalidate authentication state if fetched multiple times.

This temporary key contains 2 (or 3) things:

1. First 32 bytes: Unencrypted token needed to generate X-CSRF-XSID:
2. Next 5 bytes: RSA public key data ("e" parameter)
3. Remaining bytes: RSA public key data ("n" parameter)

Using some compatible RSA library, encrypt above 32-byte token with the given RSA key. Base64-encode the result and set it to X-CSRF-XSID: header and I was good to go.

Doing "compatible" encryption took some time and openssl didn't work out for me. I ended up grabbing actual JavaScript files (rsa.js and its dependencies) out from GS310TP and run it locally with nodejs.

GS310TP also does tricky query parameter validation with "aj4=", "dummy=", and "bj4=", but it is quite easy to workaround as you can just replay the captured string or set MD5 hash of URL parameters and append it as "bj4=" parameter.

I do have working code, but cannot share as it currently have hardcoded parameters I cannot disclose. I hope this description is good enough for you to move forward!