0

I am using aiohttp to web scrape different websites. For most websites, my code works just fine, but for https://bestbuy.com/, my code is unable to even print a response code. Does anyone know why that is?

My Code

import asyncio
import aiohttp

head = {"User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'}

async def myDriver():
    await httpBestBuy("https://www.amazon.com/")
    await httpBestBuy("https://www.bestbuy.com/")

async def httpBestBuy(URL):
    async with aiohttp.ClientSession(headers = head) as session:
        async with session.get(URL) as page:
            print(page.status)

asyncio.run(myDriver())

Output

200

I'm just trying to understand why it doesn't throw any errors for Best Buy; it just doesn't print out anything. Thank you in advance for the help :)

Notes

  • Python 3.7.9
  • Aiohttp 3.6.3
  • I have tried with headers and without headers, there is no difference
justinaylmao
  • 21
  • 1
  • some portals have biger money to create better system to recognize scripts/bots. Some portals may even keep requests open - without any answer - to block script/bots They may check something more than only `User-Agent`. – furas Dec 10 '20 at 10:24
  • code works for me only when I use `User-Agent` from my web browser: `"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0"`. It doesn't work with your `User-Agent` – furas Dec 10 '20 at 10:33

0 Answers0