2

I am having an issue with urllib on python3 that I dont know how to solve. this is my code :

import urllib.request
       hdr = { 'User-Agent' : 'super happy flair bot by /u/spladug' }
       req = urllib.request.Request(url,headers=hdr)
       data = urllib.request.urlopen(req,timeout=10).read()
       return data

Stuff I tried:

  1. Changing user agent
  2. Adding and removing timeouts
  3. curl these sites

but still there are some sites(that ends with image file extension like http://xxx.xxx/aaa.png or .jpg) that I just can't get a response, and if I am setting a timeout I am getting TimeOutError. but if i open the same site in the chrome everything working fine. Does someone have any solution or faced the issue?

site example - https://sgfm.elcorteingles.es/SGFM/dctm/MEDIA03/202006/24/00117731276964____5__210x210.jpg

lby
  • 71
  • 1
  • 5
  • It's going to be tough for anyone to troubleshoot this without at least one example of a URL that causes the problem. Try to post code problems as reproducible examples: https://stackoverflow.com/help/minimal-reproducible-example – CrazyChucky Jul 17 '20 at 14:44
  • @CrazyChunky whoops ,forgot to add the site example https://sgfm.elcorteingles.es/SGFM/dctm/MEDIA03/202006/24/00117731276964____5__210x210.jpg – lby Jul 17 '20 at 14:54
  • getting this message UP – lby Jul 17 '20 at 17:09
  • I don't have a solution, but things to add to the "doesn't work" pile: copying the entire set of headers sent by Chrome when successfully accessing the URL, checking to see if there's a redirect going on (there isn't), and running `curl` with `-k` just in case there's some kind of SSL error. (And trying `requests` too, though I see you mentioned that in your commend on Andrej's answer.) I don't know why it's not working, but if you edit your question with more detail on what's *not* working, it might help draw the right person's eye. – CrazyChucky Jul 19 '20 at 20:26

1 Answers1

0

I don't know what's the error in your code, but I'm using requests and I never had that problem.

import requests
response=requests.get(URL)
print(response.text)

How to use Headers in requests .get(): Using headers with the Python requests library's get method

Andrej
  • 2,743
  • 2
  • 11
  • 28
  • I dont recieve any error, since I don't even get the response. I know how to use requests and urllib ,It's just some specific sites that not working – lby Jul 17 '20 at 14:37