Web scraping the Uber price estimator page using Python

Question

So I'm really new to web scraping and I want to build a bot that checks the Uber ride price from point A to point B over a period of time. I used the Selenium library to input the pickup location and the destination and now I want to scrape the resulting estimated price from the page.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

# Initialize webdriver object
firefox_webdriver_path = '/usr/local/bin/geckodriver'
webdriver = webdriver.Firefox(executable_path=firefox_webdriver_path)

webdriver.get('https://www.uber.com/global/en/price-estimate/')

time.sleep(3)

# Find the search box
elem = webdriver.find_element_by_name('pickup')
elem.send_keys('name/of/the/pickup/location')

time.sleep(1)
elem.send_keys(Keys.ENTER)

time.sleep(1)
elem2 = webdriver.find_element_by_name('destination')
elem2.send_keys('name/of/the/destination')

time.sleep(1)
elem2.send_keys(Keys.ENTER)
time.sleep(5)

elem3 = webdriver.find_element_by_class_name('bn rw bp nk ih cr vk')
print(elem3.text)

Unfortunately, there's an error:

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .bn rw bp nk ih cr vk

And I can't seem to figure out a solution. While inspecting the page, the name of the class that holds the price has the following name "bn rw bp nk ih cr vk" and after some searches I found out this might be Javascript instead of HTML. (I would also point out that I'm not really familiar with them.)

Inspecting the page

Eventually, I thought I could use BeautifulSoup and requests modules and faced another error.

import requests
from bs4 import BeautifulSoup
import re
import json

response = requests.get('https://www.uber.com/global/en/price-estimate/')
print(response.status_code)

I also tried changing the User Agent in hope of resolving this HTTP error message, but it did not work. I have no idea how to approach this.

Does this answer your question? [Selenium Compound class names not permitted](https://stackoverflow.com/questions/37771604/selenium-compound-class-names-not-permitted) — Jeremy, Jun 30 '22 at 15:01
Oh yesss, this worked: `elem3 = webdriver.find_element_by_xpath("//*[@class='vl fw al ro r2']")`. Thanks a lot for the quick answer! — ttendril, Jun 30 '22 at 15:21
string `bn rw bp nk ih cr vk'` is not single class but many classes and selenium may need use dot between classes - like in CSS selection - `bn.rw.bp.nk.ih.cr.vk` — furas, Jun 30 '22 at 23:22
when I check place with price then it has different class - server may generate random names for different users, different browsers, different divices. You have to find better method to find price. I see ` — furas, Jun 30 '22 at 23:28

score 1 · Answer 1 · answered Jun 30 '22 at 15:28

Not exactly what you need. But I recently made a similar application and I will provide part of the function that you need. The only thing you need is to get the latitude and longitude. I used google_places provider for this, but iam sure there is many free services for this.

import requests
import json


def get_ride_price(origin_latitude, origin_longitude, destination_latitude, destination_longitude):
    url = "https://www.uber.com/api/loadFEEstimates?localeCode=en"

    payload = json.dumps({
      "origin": {
        "latitude": origin_latitude,
        "longitude": origin_longitude
      },
      "destination": {
        "latitude": destination_latitude,
        "longitude": destination_longitude
      },
      "locale": "en"
    })
    headers = {
      'content-type': 'application/json',
      'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
      'x-csrf-token': 'x'
    }

    response = requests.request("POST", url, headers=headers, data=payload)
    result = [[x['vehicleViewDisplayName'], x['fareString']] for x in response.json()['data']['prices']]
    return result


print(get_ride_price(51.5072178, -0.1275862, 51.4974948, -0.1356583))

OUTPUT:

[['Assist', '£13.84'], ['Access', '£13.84'], ['Green', '£13.86'], ['UberX', '£14.53'], ['Comfort', '£16.02'], ['UberXL', '£17.18'], ['Uber Pet', '£17.77'], ['Exec', '£20.88'], ['Lux', '£26.32']]

Web scraping the Uber price estimator page using Python

1 Answers1