31

Using requests I am creating an object which is in .csv format. How can I then write that object to a DataFrame with pandas?

To get the requests object in text format:

import requests
import pandas as pd
url = r'http://test.url' 
r = requests.get(url)
r.text  #this will return the data as text in csv format

I tried (doesn't work):

pd.read_csv(r.text)
pd.DataFrame.from_csv(r.text)
jezrael
  • 822,522
  • 95
  • 1,334
  • 1,252
sparrow
  • 10,794
  • 12
  • 54
  • 74
  • 2
    difficult to answer without seeing data. – shivsn Aug 29 '16 at 19:25
  • May be you need save the response data to a file and check the file content. Then read the file to csv, check if this approach works. If not then there is something wrong in the data – Shijo Aug 29 '16 at 19:27
  • 1
    http://stackoverflow.com/questions/32400867/pandas-read-csv-from-url/32401251#32401251, no need for requests unless you are posting some data that allows you to access the content – Padraic Cunningham Aug 29 '16 at 19:34
  • @PadraicCunningham, I think you are wrong about requests, Its urllib2 or request. shhh, urllib2 has security flaws. which allow file access. So, requests is safer. – Merlin Aug 29 '16 at 19:43
  • @Padraic, I omitted this part in my question but I needed to include a special header format in my request which is why I used requests instead of importing the url directly. headers = {'user1': 'AppInterface'} – sparrow Aug 29 '16 at 19:49

4 Answers4

62

Try this

import requests
import pandas as pd
import io

urlData = requests.get(url).content
rawData = pd.read_csv(io.StringIO(urlData.decode('utf-8')))
Merlin
  • 24,552
  • 41
  • 131
  • 206
  • can we convert dataframe output to URL. display dataframe table output in the form of url : any one can access it and see the dataframe data by opening that url ? – Ravi Oct 31 '20 at 18:47
23

I think you can use read_csv with url:

pd.read_csv(url)

filepath_or_buffer : str, pathlib.Path, py._path.local.LocalPath or any object with a read() method (such as a file handle or StringIO)

The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. For instance, a local file could be file ://localhost/path/to/table.csv

import pandas as pd
import io
import requests

url = r'http://...' 
r = requests.get(url)  
df = pd.read_csv(io.StringIO(r))

If it doesnt work, try update last line:

import pandas as pd
import io
import requests

url = r'http://...' 
r = requests.get(url)  
df = pd.read_csv(io.StringIO(r.text))
jezrael
  • 822,522
  • 95
  • 1,334
  • 1,252
5

Using "read_csv with url" worked:

import requests, csv
import pandas as pd
url = 'https://arte.folha.uol.com.br/ciencia/2020/coronavirus/csv/mundo/dados-bra.csv'
corona_bra = pd.read_csv(url)
print(corona_bra.head())
Bento
  • 51
  • 1
  • 4
  • I guess this will depend on the request URL, I need to pass parameters to get(), for example, requests.get(url,params=dict()) – gudé Jan 29 '22 at 11:57
  • @gudé You can simply convert the parameters to a string using `'&'.join([f'{key}={value}' for key, value in params.items()])` and append to the rest of the URL. – Keivan Ipchi Hagh May 20 '22 at 09:10
2

if the url has no authentication then you can directly use read_csv(url)

if you have authentication you can use request to get it un-pickel and print the csv and make sure the result is CSV and use panda.

You can directly use importing import csv

rkoots
  • 199
  • 5
  • 4