1

I am trying to scrape heading from an Amazon listing. But it seems there is some error in my proxy. Here is the code I've tried:

import requests
from bs4 import BeautifulSoup
url="https://www.amazon.com/Kindle-Paperwhite-Essentials-Bundle-including/dp/B0898ZK226/ref=sr_1_3?dchild=1&keywords=kindle&qid=1597051671&sr=8-3"

headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"}
proxyDict = { 
             "http": "http://10.10.1.10:3128",
             "https": "http://10.10.1.10:1080",
             "ftp"   : "ftp://10.10.1.10:3128"
           }
page = requests.get(url, headers=headers, proxies=proxyDict) #Error here.
print(page.status_code)
soup = BeautifulSoup(page.content, "html.parser")
title = soup.find(id = "productTitle")
if title:
   title = title.get_text().strip()
else:
   title = "Title: Error 404"
print(title)

Without a proxy, Code shows None in output.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
  • Hi there, you can check out this post [here](https://stackoverflow.com/questions/38785877/spoofing-ip-address-when-web-scraping-python/56654164#56654164), when it comes to using proxies while webscarping. – Yannik Suhre Aug 10 '20 at 10:19
  • first check if page doesn't use JavaScript to add item. `requests` and `BS` can't run JavaScript and `proxy` can't change it. – furas Aug 10 '20 at 11:35
  • shouldn't be `IP:PORT` without `http://` and `ftp://` ? – furas Aug 10 '20 at 11:36

1 Answers1

2

Previous Code wasn't able to scrape JavaScript. Here is the correct code:

from bs4 import BeautifulSoup
import requests
from requests_html import HTMLSession
# create an HTML Session object
session = HTMLSession()
# Use the object above to connect to needed webpage
resp = session.get("https://www.amazon.com/Sceptre-E248W-19203R-Monitor-Speakers-Metallic/dp/B0773ZY26F/ref=sr_1_2?crid=1861TM8A5NDPX&dchild=1&keywords=monitors&qid=1597071906&sprefix=monitors%2Caps%2C364&sr=8-2")
# Run JavaScript code on webpage
resp.html.render()
soup = BeautifulSoup(resp.html.html, "lxml")
title = soup.find(id = "productTitle").get_text().strip()
print (title)
DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22