-1

as an example, i have a super-market website and there s a section showing market's stores. for choosing locations a dropdown list exists. What i am trying to do is based on my choice from dropdown list i want to get quantities of stores (blue frame). here s picture;

enter image description here

I accomplished getting values from dropdown list by this code:

import requests
from bs4 import BeautifulSoup

url="https://www.migros.com.tr/en-yakin-migros"

r=requests.get(url)
ht=r.content
soup=BeautifulSoup(ht,"html.parser")


soup= soup.find("div",class_="stores-selection-container stores-city-select address-part")
items=soup.select("option[value]")

#values=[item.get("value") for item in items]

cities=[item.text for item in items]
del cities[0] #first index is empty and removed

after that, I am stucked. what I want is telling the computer to select the city (from citys list) from dropdown and then getting number (blue frame)

I'd be grateful if you tell me the path that i need to take.

bibakcamki
  • 83
  • 7

1 Answers1

1

Here is need to requests post request with data. Data collect from the dev network tap.

import requests
from bs4 import BeautifulSoup

# url="https://www.migros.com.tr/en-yakin-migros"

# r=requests.get(url)

data1 = {"cityName": "İSTANBUL",
"townName": "ADALAR",
"cityId": '', 
"townId": "34001"}
# 1 Mağaza Listeleniyor


# cityName: İSTANBUL
# townName: BAĞCILAR
# cityId: 34
# townId: 34025
    
data = {"cityName": "İSTANBUL",
"townName": "BAĞCILAR",
"cityId": '34', 
"townId": "34025"} 
# 11 Mağaza Listeleniyor

# cityName: İZMIR
# townName: ÇIĞLI
# cityId: 35
# townId: 35025

# 15 Mağaza Listeleniyor    

# cityName: İSTANBUL
# townName: ADALAR
# cityId: 
# townId: 34001

# cityName: İSTANBUL
# townName: ADALAR
# cityId: 
# townId: 34001

post_url = "https://www.migros.com.tr/stores"

response = requests.post(post_url, data=data)
print(response.status_code)
print()
soup=BeautifulSoup(response.content,"html.parser")
print(soup)

# soup= soup.find("div",class_="stores-selection-container stores-city-select address-part")
# items=soup.select("option[value]")
# items
# print('soup')

#values=[item.get("value") for item in items]

# cities=[item.text for item in items]
# cities
# del cities[0] #first index is empty and removed

Hopes it help you.

Samsul Islam
  • 2,581
  • 2
  • 17
  • 23
  • 1
    As I understood, 1st i'll gather values from dropdown list > 2nd i'll format them as you show me "-data- example"> 3rd I ll make a request with "post" method and get how i want ?? – bibakcamki Jan 12 '21 at 21:06
  • 1
    yes, You can follow it https://stackoverflow.com/questions/8550114/can-scrapy-be-used-to-scrape-dynamic-content-from-websites-that-are-using-ajax/17697329 – Samsul Islam Jan 13 '21 at 06:38