Python Scraping to extract names of vessels

Question

from scrapy import Selector
import requests
url = 'http://lines.coscoshipping.com/home/Services/ship/0'
html = requests.get(url).content
sel = Selector(text = html)
sel.xpath('//tr/td[1]/div/div/div[2]/div[1]/text()').extract()

'''Can anyone help with this scraping, and I just want to extract the names of each vessels. Many thanks in advance'''

Answer 1

The following script should produce the results accordingly.

import requests

link = 'http://lines.coscoshipping.com/homeapi/ship/findShips.do?slots=0&language=1'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
    s.headers['Referer'] = 'http://lines.coscoshipping.com/home/Services/ship/0'
    r = s.get(link)
    for item in r.json()['data']['content']:
        print(item['shipNameCn'],item['shipNameEn'])

Output are like:

中海太平洋 CSCL PACIFIC OCEAN
中海印度洋 CSCL INDIAN OCEAN
中海大西洋 CSCL ATLANTIC OCEAN
中海之星 CSCL STAR
中海土星 CSCL SATURN
中海天王星 CSCL URANUS
中海水星 CSCL MERCURY
中海木星 CSCL JUPITER
中海金星 CSCL VENUS
中海火星 CSCL MARS
中海海王星 CSCL NEPTUNE
中海阿里山 JEBEL ALI

Answer 2

The problem:

Inspecting the HTML of your shown URL, it appears that the page content is mostly loaded dynamically. You can use a tool that can run the JavaScript to generate the page content so you can then extract the relevant information.

The requests library won't do this for you. You can instead use the selenium library.

Using Selenium:

Firstly, observe that the HTML for the vessel names looks like this:

<div data-v-5c859f2b="" class="names">
    <div data-v-5c859f2b="">中海太平洋</div>
    <div data-v-5c859f2b="">CSCL PACIFIC OCEAN</div>
</div>

The code below find_elements_by_class_name() to extract the HTML tags with the class names (which is used for the the vessel names).

Then, find_elements_by_tag_name() is used to find the child div tags, which contain the Chinese and English names.

from selenium import webdriver

import textwrap


url = 'http://lines.coscoshipping.com/home/Services/ship/0'
driver = webdriver.Firefox(executable_path='YOUR PATH')  # or Chrome
driver.get(url)
for vessel in driver.find_elements_by_class_name('names'):
    chinese, english = vessel.find_elements_by_tag_name('div')
    print(textwrap.dedent(f'''
        Chinese: {chinese.text}
        English: {english.text}
    '''))

I've also used textwrap.dedent() to prettify the output.

Example output:

Chinese: 中海太平洋
English: CSCL PACIFIC OCEAN


Chinese: 中海印度洋
English: CSCL INDIAN OCEAN


Chinese: 中海大西洋
English: CSCL ATLANTIC OCEAN


Chinese: 中海之星
English: CSCL STAR

...

See also this post about how to download a driver (either Chrome or Firefox) and add it to the $PATH.

An alternative way:

Using splitlines(), we can extract the Chinese and English names of the vessels more succinctly from each of the divs:

from selenium import webdriver

import textwrap


url = 'http://lines.coscoshipping.com/home/Services/ship/0'
driver = webdriver.Firefox(executable_path='YOUR PATH')  # or Chrome
driver.get(url)
for vessel in driver.find_elements_by_class_name('names'):
    chinese, english = vessel.text.splitlines()
    print(textwrap.dedent(f'''
        Chinese: {chinese}
        English: {english}
    '''))

This is a bit more presumptive, but often does work (as in this case).