How to convert url data to csv using python

Question

i am trying to download the data from the following url and tying to save it as csv data but the output i am getting is a text file. can anyone pls help what i am doing wrong here ? also, is it possible to add multiple url in the same script and download multiple csv files.

import csv

import pandas as pd
import requests
from datetime import datetime

CSV_URL = ('https://dsv-ops-toolkit.ihsmvals.com/ftp?config=fenics-bgc&file=IRSDATA_20211129_1700_Intra.csv&directory=%2FIRS%2FIntraday%2FDaily')

with requests.Session() as s:
    download = s.get(CSV_URL)

    decoded_content = download.content.decode('utf-8')

    cr = csv.reader(decoded_content.splitlines(), delimiter=',')
    date =datetime.now().strftime('%y%m%d')
    my_list = list(cr)
    df=pd.DataFrame(my_list)
    df.to_csv(f'RFR_{date}')

The URL doesn't open. But you might want to try using `df.to_csv(f"RFR_{date}.csv")` i.e. specify the ".csv" extension in the file name — not_speshal, Nov 29 '21 at 17:56
check this [link](https://stackoverflow.com/questions/46410307/convert-a-list-of-data-from-url-to-csv-in-python?rq=1) this could help you — SmatIsNice, Nov 29 '21 at 17:58
amazing, it worked. can you please answer the other part as well. how can i add one more url in the same script and download two csv files running the same script — Rahul Vaidya, Nov 29 '21 at 17:59

score 0 · Answer 1 · answered Dec 03 '21 at 18:27

You can create a list of your necessary URLs like:

urls = ['http://url1.com','http://url2.com','http://url3.com']

Iterate through the list for each url and your requests will be as it is:

for each_url in urls:
    with requests.Session() as s:
        # your_code_here

Hope you'll find this helpful.

How to convert url data to csv using python

1 Answers1