Scraping data from marinetraffic page

Question

I am tring to scrape data from this webpage : marine traffic

I did try normal scraping in python and Selenium but I can't figure out any of the target data. (latitude/longitude/speed)

Is there a special format that I am missing ?

This is the code I started with

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless') 
driver = webdriver.Chrome("C:/webdrivers/chromedriver.exe", options=options)
page = driver.page_source

But by making a simple search of text with CTRL + F I can't find anything satisfying.

Any idea of how to scrape it down ?

Thanks

Answer 1

If you view the page in a browser, and log your browser's network traffic, you'll notice some XHR HTTP GET requests being made to various API endpoints, the response of which is JSON and contains the information you're looking for. All you have to do is imitate those requests - no BeautifulSoup or Selenium required:

def get_ship_position(ship_id):
    import requests

    url = "https://www.marinetraffic.com/vesselDetails/latestPosition/shipid:{}".format(ship_id)

    headers = {
        "accept": "application/json",
        "accept-encoding": "gzip, deflate",
        "user-agent": "Mozilla/5.0",
        "x-requested-with": "XMLHttpRequest"
    }

    response = requests.get(url, headers=headers)
    response.raise_for_status()

    return response.json()


def main():

    from datetime import datetime

    data = get_ship_position("371441")
    ts = datetime.utcfromtimestamp(data["lastPos"])
    print("Last known position: {} / {} @ {}".format(data["lat"], data["lon"], ts))
    
    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

Output:

Last known position: -1.53057 / -48.77838 @ 2021-08-04 10:33:33
>>> 

Answer 2

First, for using Selenium in headless mode, you have to define screen size

options.add_argument('--window-size=1920,1080')

To get the coordinates and speed you can use this:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 20)

coordinates = wait.until(EC.visibility_of_element_located((By.XPATH, "//p[contains(text(),'Latitude')]/b"))).text

speed =  wait.until(EC.visibility_of_element_located((By.XPATH, "//p[contains(text(),'Speed')]/b"))).text

Also, since you are using headless mode, these settings may be usefull

options.add_argument('--no-sandbox')
options.add_argument('----disable-dev-shm-usage')

Answer 3

There are few things,

1. You will need to click on Accept cookies button.
2. You will need to click on X button which is visible sometime and sometime don't.
3. You need explicit waits as well.

Sample code :

options = webdriver.ChromeOptions()
options.add_argument("--disable-infobars")
options.add_argument("--start-maximized")
options.add_argument("--disable-extensions")
options.add_experimental_option("prefs", {"profile.default_content_setting_values.notifications": 2})
options.add_argument('--window-size=1920,1080')
options.add_argument("--headless")
options.add_experimental_option("prefs", {"profile.default_content_settings.cookies": 2})
driver = webdriver.Chrome(options = options)
driver.implicitly_wait(30)
driver.maximize_window()
driver.get("https://www.marinetraffic.com/en/ais/details/ships/shipid:371441/mmsi:310554000/imo:9312456/vessel:STENA_PERROS")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='AGREE']"))).click()
try:
    if(len(driver.find_elements(By.XPATH, "//*[name()='svg' and @class='MuiSvgIcon-root']/ancestor::button[contains(@class,'jss17')]"))) >0:
        print("X is visible")
        wait.until(EC.visibility_of_element_located((By.XPATH, "//*[name()='svg' and @class='MuiSvgIcon-root']/ancestor::button[contains(@class,'jss17')]"))).click()
        print("done clicking")
    else:
        print("X was not visible")
except:
    print("something went wrong")
    pass

print(wait.until(EC.visibility_of_element_located((By.XPATH, "//b//a[contains(@href,'/en/ais/hom')]"))).text)
print(wait.until(EC.visibility_of_element_located((By.XPATH, "//b//a[contains(@href,'/en/ais/hom')]/ancestor::p/following-sibling::p/b"))).text)
print(wait.until(EC.visibility_of_element_located((By.XPATH, "//b//a[contains(@href,'/en/ais/hom')]/ancestor::p/following-sibling::p[2]/b"))).text)

Imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output :

X is visible
done clicking
-1.53057° / -48.77838°
Underway using Engine
1.7 kn / 250 °

Process finished with exit code 0