How to download a file from GitHub using Python

Question

I don't know if it was posted before but I could not find it neither on Stackoverflow nor on the rest of the web

I'm working on a project where we need a live dataset so we can update our database everyday. I found a GitHub repository where a csv file is updated everyday and I need to download it to my local as I run a code. How am I supposed to do that ?

We are using Python and PostgreSQL

BlackMath · Accepted Answer · 2021-09-30T13:33:40.880

2

find the path of your desidered csv (raw format):

csv_url = 'https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/latest/owid-covid-latest.csv'

Then read it in python with :

df = pd.read_csv(csv_url, error_bad_lines=False)
print(df)

You can now update your sql db with:

df.to_sql('my_SQL_table', con=my_engine, if_exists='replace')

edited Sep 30 '21 at 13:33

answered Sep 30 '21 at 13:16

BlackMath

1,708
1
11
14

why to use panadas? – balderman Sep 30 '21 at 13:21
OP says "so we can update our database everyday".... Importing a csv with pandas (as df) will let you to easily use `pandas.DataFrame.to_sql ` to manage a sql update. – BlackMath Sep 30 '21 at 13:29

score 0 · Answer 2 · answered Sep 30 '21 at 13:20

There is already an answer for your question, check this link below:
How to download a file from Github using Requests

If you want to download just the file, you need to use the Raw link at the top of the page, which will be (for your example):

https://raw.github.com/someguy/brilliant/master/somefile.txt
Note the change in domain name, and the blob/ part of the path is gone.

Try this code to download the COVID-2019 20-20ECDC 20(2020).csv file:

>>> import requests
>>> r = requests.get('https://github.com/owid/covid-19-data/blob/master/public/data/ecdc/COVID-2019%20-%20ECDC%20(2020).csv')
>>> 'Requests:' in r.text
True
>>> r.headers['Content-Type']
'text/html; charset=utf-8'
>>> r = requests.get('https://github.com/owid/covid-19-data/blob/master/public/data/ecdc/COVID-2019%20-%20ECDC%20(2020).csv')
>>> 'Requests:' in r.text
True
>>> r.headers['Content-Type']
'text/plain; charset=utf-8'

score 0 · Answer 3 · answered Sep 30 '21 at 13:22

You can automate this by using the request library and downloading the CSV file using the link to the file from github. As long as its file that it is the file that is being updated every time this runs it should be the most recent data.

import requests

url = 'https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/owid-covid-data.csv'
r = requests.get(url, allow_redirects=True)

open('data.csv', 'wb').write(r.content)

How to download a file from GitHub using Python

3 Answers3