-1

I want to extract the info from each restaurant from this webpage: https://orderappetit.com/restaurants-search

import requests
from bs4 import BeautifulSoup

url = 'https://orderappetit.com/restaurants-search'

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) '\
       'AppleWebKit/537.36 (KHTML, like Gecko) '\
       'Chrome/75.0.3770.80 Safari/537.36'}

response = requests.get(url,headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

restaurants = soup.find(class_='ant-space-item')
print(restaurants)

Output

[]

I would like to know why my statement find is not working and how can I know if I need to use select or find

coding
  • 917
  • 2
  • 12
  • 25

1 Answers1

0

The page is loaded dynamically. There is nothing in the static html. You have to fetch the data from the api.

It's up to you to figure out the paramters you need by looking in the Dev Tools:

import requests
import pandas as pd

url = 'https://api.orderappetit.com/api/customers/stores'
payload ={
    'page': '1',
    'limit': '12',
    'location': '609b7a404575ff2493d02d54',
    'isRandom': '1',
    'type': ''
 }


jsonData = requests.get(url, params=payload).json()
stores = jsonData['stores']

df = pd.DataFrame(stores)

Output:

print(df['name'])
0             Orazio's by Zarcone 
1     Carbone's Pizza | South Park
2                       Colter Bay
3                   Saigon Bangkok
4      Marco’s Italian Restaurant 
5                   Lakeshore Cafe
6       Cookies and Cream | Seneca
7         Deep South Taco | Hertel
8            Macy's Place Pizzeria
9                  Wise Guys Pizza
10                     Tappo Pizza
11      Aguacates Mex Bar & Grill 
Name: name, dtype: object

Or to search for specific food and choose dilery/pickup:

search = 'pizza'

url = 'https://api.orderappetit.com/api/customers/stores/search'
payload ={
'generic': search,
'type': 'delivery',
'page': '1',
'limit': '8',
'location': '609b7a404575ff2493d02d54'}

jsonData = requests.get(url, params=payload).json()
stores = jsonData['stores']

df = pd.json_normalize(jsonData, record_path=['stores'])

Output:

print(df['store.name'])
0                Macy's Place Pizzeria
1                     Prohibition 2020
2        Trattoria Aroma Williamsville
3       Sidelines Sports Bar and Grill
4          Marco’s Italian Restaurant 
5                 Orazio's by Zarcone 
6    Butera's Craft Beer & Craft Pizza
7                      Allentown pizza
Name: store.name, dtype: object

enter image description here

chitown88
  • 27,527
  • 4
  • 30
  • 59
  • :O where did you find the url for the API? – coding Apr 14 '22 at 15:32
  • 1
    Go to the Dev Tools (ctrl-shift-i). Then look in the Network -> XHR. You'll see the requests pop up in the panel (might need to refresh the page). You then just search through those to find where the data is. Then look at Headers to find the url. – chitown88 Apr 14 '22 at 15:35
  • 1
    then go to payload tab to see what you need from there. See pic in solution – chitown88 Apr 14 '22 at 15:36