0

I am trying to get trip times using beautiful soup:

https://betaplanner.trimet.org/map/#/?fromPlace=10255%20SW%20CANYON%20RD%3A%3A45.493227%2C-122.782138&toPlace=4809%20N%20KERBY%20AVE%3A%3A45.557817%2C-122.673216&date=2019-08-14&time=16%3A56&arriveBy=false&mode=BUS%2CTRAM%2CRAIL%2CGONDOLA%2CCAR_RENT&showIntermediateStops=true&optimize=QUICK&ignoreRealtimeUpdates=true&companies=NaN&minTransitDistance=50%25&searchTimeout=10000&onlyTransitTrips=true&ui_activeItinerary=0

It seems like the problem is related to javascript rendering, however, when I tried doing html-requests, I have gotten an error

For Bs4:

 r = session.get(*linkfromabove*)
    soup = BeautifulSoup(r.content, 'html.parser')
    soup.select

For HTML_Requests:

import requests
from requests_html import HTMLSession
session = HTMLSession()
r = session.get(linkfromabove)
r.html.render(wait=8, sleep=8)

Actual Results using BS4:

<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="ie=edge" http-equiv="x-ua-compatible"/>
<title>TriMet MOD OTP Demo</title>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="https://fonts.googleapis.com/css? 
family=Hind:300,400,500,600,700" 
rel="stylesheet"/>
<link href="main.css" rel="stylesheet"/></head>
<body>
<div id="main"></div>
<script src="bundle.js" type="text/javascript"></script></body>
</html>

From Html-Requests:

 pyppeteer.errors.NetworkError: Protocol error Target.closeTarget: Target closed.
Christian
  • 4,902
  • 4
  • 24
  • 42
eveng
  • 1

1 Answers1

1

If you inspect network tab you will see a request that returns json with your info

import requests

r = requests.get('https://maps.trimet.org/otp_mod/plan?fromPlace=10255%20SW%20CANYON%20RD%3A%3A45.493227%2C-122.782138&toPlace=4809%20N%20KERBY%20AVE%3A%3A45.557817%2C-122.673216&date=2019-08-14&time=16%3A56&arriveBy=false&mode=BUS%2CTRAM%2CRAIL%2CGONDOLA%2CCAR_RENT%2CWALK&showIntermediateStops=true&optimize=QUICK&ignoreRealtimeUpdates=true&companies=NaN&minTransitDistance=50%25&searchTimeout=10000&onlyTransitTrips=true').json()
QHarr
  • 83,427
  • 12
  • 54
  • 101
  • did you copy exactly as is? Seems to still work for me. – QHarr Aug 23 '19 at 23:48
  • see if you can find the same url in the network tab when you refresh the page with F5 – QHarr Aug 23 '19 at 23:49
  • Thanks for quick response. When I tried that I got this error code: JSONDecodeError: Expecting value: line 1 column 1 (char 0), using my original link. Where did you find this new link? – eveng Aug 23 '19 at 23:50
  • In the network tab. I opened the original url and viewd in network tab F12 when then pressing F5 to refresh page. I searched the network traffic for a value I expected to find in results. [Example](https://stackoverflow.com/questions/56277464/how-to-import-a-table-from-web-page-with-div-class-to-excel/56279841#56279841) and [example](https://stackoverflow.com/questions/56923231/is-there-a-way-to-slow-down-a-web-scraper-so-it-will-pick-up-the-code/56924071#56924071) – QHarr Aug 24 '19 at 07:49