-1

I want to scrape product names and prices from the website : https://www.carrefouruae.com/mafuae/en/c/F21600000

import requests
html = requests.get('https://www.carrefouruae.com/mafuae/en/c/F21600000')
soup = BeautifulSoup(html.content, "html5lib")
soup.findAll('ul',attrs={'class':'css-1wgjvs'})

It's returning an empty list. It's unable to fetch the actual page source with the product names. What is the reason? How can I fetch the product details from the site?

Mervyn Ludick
  • 120
  • 8
Monika
  • 1
  • 1
    call the [API](https://www.carrefouruae.com/api/v8/categories/F21600000?filter=&sortBy=relevance&currentPage=1&pageSize=60&maxPrice=&minPrice=&areaCode=Dubai%20Festival%20City%20-%20Dubai&lang=en&displayCurr=AED&latitude=25.2171003&longitude=55.3613635&nextOffset=&needVariantsData=true&requireSponsProducts=true&responseWithCatTree=true&depth=3) – αԋɱҽԃ αмєяιcαη May 25 '23 at 13:25

1 Answers1

2
import requests
import json


headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/113.0',
    'appId': 'Reactweb',
    'storeId': 'mafuae',
}


def main():
    with requests.session() as req:
        req.headers.update(headers)
        params = {
            "areaCode": "Dubai Festival City - Dubai",
            "currentPage": "0",
            "depth": "3",
            "displayCurr": "AED",
            "filter": "",
            "lang": "en",
            "latitude": "25.2171003",
            "longitude": "55.3613635",
            "maxPrice": "",
            "minPrice": "",
            "needVariantsData": "true",
            "nextOffset": "",
            "pageSize": "60",
            "requireSponsProducts": "true",
            "responseWithCatTree": "true",
            "sortBy": "relevance"
        }
        r = req.get(
            'https://www.carrefouruae.com/api/v8/categories/F21600000', params=params)
        with open('data.json', 'w', encoding='utf-8-sig') as f:
            json.dump(r.json(), f, indent=4)


if __name__ == "__main__":
    main()
αԋɱҽԃ αмєяιcαη
  • 11,825
  • 3
  • 17
  • 50
  • 1
    dear αԋɱҽԃ αмєяιcαη - awesome you allway provide great solutions: i have learned alot from you - at the moment i struggle to get get data out of clutch.co: with BS4 and requests failed: - i cannot run all the solutions on colab - what should i do now https://stackoverflow.com/questions/76362600/getting-data-out-of-clutch-co-with-bs4-and-requests-failed - can you help here - thanks in advance!! You rock!! keep up the great work ... – malaga Jun 06 '23 at 09:30
  • 1
    and see also here https://stackoverflow.com/questions/76409097/driver-webdriver-chrome-issues-with-a-selenium-approach-how-to-work-aro i am trying to figure out - can you help here - thanks in advance!! You rock!! keep up the great work – malaga Jun 06 '23 at 09:54