Cannot read a csv with urls to web scrape them in python

Question

I am fresh new to python so I tried with visual studio and windows 7 the following

import csv
from bs4 import BeautifulSoup 
import requests 

contents = []
with open('websupplies.csv','r') as csvf: # Open file in read mode
   urls = csv.reader(csvf)

   for url in urls:
      contents.append(url) # Add each url to list contents


for url in contents:  # Parse through each url in the list.
   page = requests.get(url).content
   soup = BeautifulSoup(page, "html.parser")

   price = soup.find('span', attrs={'itemprop':'price'})
   availability = soup.find('div', attrs={'class':'product-availability'})

but I get - No connection adapters were found for .. '['a url']'

why?

The structure of csv is like the following

https://www.websupplies.gr/epeksergastis-intel-core-i5-8400-9mb-2-80ghz-bx80684i58400
https://www.websupplies.gr/epeksergastis-intel-celeron-g3930-2mb-2-90ghz-bx80677g3930
https://www.websupplies.gr/epeksergastis-amd-a6-9500-bristol-ridge-dual-core-3-5ghz-socket-am4-65w-ad9500agabbox

they dont have at the end a semicolumn

Possible duplicate of [Python Requests - No connection adapters](https://stackoverflow.com/questions/15115328/python-requests-no-connection-adapters) — mad_, Mar 12 '19 at 14:24

score 2 · Accepted Answer · answered Mar 12 '19 at 14:53

Your file is a flat list of URLs. It's not really a CSV.

The CSV reader reads each row into its own list. So the structure of the loaded data would be:

[
  ["https://www.websupplies.gr/epeksergastis-intel-core-i5-8400-9mb-2-80ghz-bx80684i58400"],
  ["https://www.websupplies.gr/epeksergastis-intel-celeron-g3930-2mb-2-90ghz-bx80677g3930"],
  ["https://www.websupplies.gr/epeksergastis-amd-a6-9500-bristol-ridge-dual-core-3-5ghz-socket-am4-65w-ad9500agabbox"],
]

One way to fix this would be to use url[0] as the parameter to requests.get, but really the proper fix is not to use CSV at all. Since you have only one piece of data per line, you can just read the data directly and pass it to requests:

with open('websupplies.csv','r') as csvf: # Open file in read mode 
   for line in csvf:
      contents.append(line.strip('\n')) # Add each url to list contents

score 1 · Answer 2 · answered Mar 12 '19 at 14:26

1

In this question it says that requests needs the http scheme, maybe it's this the problem? You also have to remove the /n when you read the lines from the file

answered Mar 12 '19 at 14:26

Giovanni Cappelletti

71
9

1

in your answer has a question, it looks like a comment. – Brown Bear Mar 12 '19 at 14:33
i know, but i don't have enough reputation, i would have written a comment. I didn't respond in fact – Giovanni Cappelletti Mar 12 '19 at 14:34

Cannot read a csv with urls to web scrape them in python

2 Answers2