Questions tagged [urlretrieve]
54 questions
32
votes
2 answers
urllib.urlretrieve with custom header
I am trying to retrieve a file using urlretrieve, while adding a custom header.
While checking the codesource of urllib.request I realized urlopen can take a Request object in parameter instead of just a string, allowing to put the header I…
realUser404
- 2,111
- 3
- 20
- 38
6
votes
2 answers
How to Download only the first x bytes of data Python
Situation: The file to be downloaded is a large file (>100MB). It takes quite some time, especially with slow internet connection.
Problem: However, I just need the file header (the first 512 bytes), which will decide if the whole file needs to be…
Timothy Wong
- 689
- 3
- 9
- 28
4
votes
1 answer
urllib.request in Python 3 - check if file is downloadable
In Python 3.8.2 I download files with:
import urllib.request
urllib.request.urlretrieve(url_address, file_name)
How can I check if file on url_address is downloadable without downloading it?
I tried with try statement. It only raises Error when it…
Łukasz Ungier
- 71
- 4
4
votes
1 answer
How to include try and Exceptions tests in a thousands downloads program that uses selenium and requests?
I have a program to download photos on various websites. Each url is formed at the end of the address by codes, which are accessed in a dataframe. In a dataframe of 8,583 lines
The sites have javascript, so I use selenium to access the src of the…
Reinaldo Chaves
- 965
- 4
- 16
- 43
4
votes
1 answer
urllib.request.urlretrieve ERROR trying to download jpeg in Python
I am trying to download a .jpg file, using urllib.request.urlretrieve(url, filename) in Python 3.5.2. The url is http://dm.victoriassecret.com/product/404x539/V603923_CROP1.jpg . The following error raises:
http.client.RemoteDisconnected: Remote…
dimosbele
- 381
- 3
- 19
3
votes
1 answer
How to read data from an online gzip file without creating a local copy?
I am new to programming and trying my hand at training an AI model with the MNIST database of handwritten digits. I already have a code that's working but now want to delve more into the details.
First thing I have to do in this project is to read…
Gvantsa
- 69
- 6
3
votes
1 answer
Should I switch from "urllib.request.urlretrieve(..)" to "urllib.request.urlopen(..)"?
1. Deprecation problem
In Python 3.7, I download a big file from a URL using the urllib.request.urlretrieve(..) function. In the documentation (https://docs.python.org/3/library/urllib.request.html) I read the following just above the…
K.Mulier
- 8,069
- 15
- 79
- 141
3
votes
2 answers
Load data from bucket google cloud
Here is a function to load data from google cloud bucket.
action_dataset_folder_path = 'action-data-set'
zip_path = 'actions.zip'
url='http://console.cloud.google.com/storage/browser/actions'
class LoadProgress(tqdm):
last_block = 0
def…
LearnToGrow
- 1,656
- 6
- 30
- 53
3
votes
1 answer
Download file with urlretrieve() to subfolder
Is it possible to use urlretrieve() in order to download something into a subfolder without expressing it in an absolute but relative manner?
For example:
urllib.request.urlretrieve(url, '/downloads/2017/foo.txt')
Everytime I add a path to the…
technical_difficulty
- 427
- 2
- 5
- 20
2
votes
0 answers
How to download images from GitLab with token) authentication using Pythons urllib.urlretrieve
I'm working on a script to persistently save GitLab Issues (in my case requirements) as Markdown files in a repository with tags and as a compiled requirements document in PDF via GitLab CI. In addition, the script needs to download all attached…
Paebbels
- 15,573
- 13
- 70
- 139
2
votes
1 answer
urlretrieve for image returns HTTP Error 403: Forbidden
Hei guys, I am trying to get an image using BeautifulSoup but I am getting an error when doing so:
Here is my code:
imgUrl = "https://www.residentadvisor.net/images/events/flyer/2017/7/no-0713-986042-front.jpg"
try:
urlretrieve(imgUrl,…
anho
- 1,705
- 2
- 20
- 38
1
vote
0 answers
In Django how to delete, retrieve, and update a table when calling an external API
I was wondering what the correct way is to delete one table in the database and update it with new information from an external API every time that is called.
Basically, whenever the API would be called, the information saved in the table should be…
David
- 97
- 7
1
vote
1 answer
Parallel downloading with urlretrieve
I regularly have to download and rename HTML pages in bulk and wrote this simple code for it a while ago:
import shutil
import os
import sys
import socket
socket.setdefaulttimeout(5)
file_read = open(my_file, "r")
lines =…
mythos917
- 11
- 2
1
vote
1 answer
Getting forbidden error from code downloading image from web
import random
import urllib.request
def download_web_image(url):
name = random.randrange(1, 1000)
full_name = str(name) + ".jpg"
urllib.request.urlretrieve(url,…
danish manzar
- 11
- 1
1
vote
1 answer
HTTPS Link using urllib.urlretrieve in Python
I'm using this code and it works fine when using HTTP but it is not working when connecting over HTTPS
import urllib
import ssl
(fn,hd) = urllib.urlretrieve('http://host.com/file.py')
execfile(fn)
Does anyone know how to connect to HTTPS using the…
ntsu
- 67
- 1
- 7