Uploading a csv type data using python request.put without reading from a saved csv file?

Question

i have an api end point where i am uploading data to using python. end point accepts

putHeaders = {
    'Authorization': user,

    'Content-Type': 'application/octet-stream' }

My current code is doing this

.Save a dictionary as csv file

.Encode csv to utf8

dataFile = open(fileData['name'], 'r').read()).encode('utf-8')

.Upload file to api end point

fileUpload = requests.put(url,
                          headers=putHeaders,
                          data=(dataFile))

What i am trying to acheive is

loading the data without saving

so far i tried

converting my dictionary to bytes using

data = json.dumps(payload).encode('utf-8')

and loading to api end point . This works but the output in api end point is not correct.

Question

Does anyone know how to upload csv type data without actually saving the file ?

Answer 1

Updated answer

I hadn't realized your input was a dictionary, you had mentioned the dictionary was being saved as a file. I assumed the dictionary lookup in your code was referencing a file. More work needs to be done if you want to go from a dict to a CSV file-like object.

Based on the I/O from your question, it appears that your input dictionary has this structure:

file_data = {"name": {"Col1": 1, "Col2": 2}}

Given that, I'd suggest trying the following using csv and io:

import csv
import io

import requests

session = requests.Session()
session.headers.update(
    {"Authorization": user, "Content-Type": "application/octet-stream"}
)

file_data = {"name": {"Col1": 1, "Col2": 2}}

with io.StringIO() as f:
    name = file_data["name"]
    writer = csv.DictWriter(f, fieldnames=name)
    writer.writeheader()
    writer.writerows([name])  # `data` is dict but DictWriter expects list of dicts

    response = session.put(url, data=f)

You may want to test using the correct MIME type passed in the request header. While the endpoint may not care, it's best practice to use the correct type for the data. CSV should be text/csv. Python also provides a MIME types module:

>>> import mimetypes
>>> 
>>> mimetypes.types_map[".csv"]
'text/csv'

---

Original answer

Just open the file in bytes mode and rather than worrying about encoding or reading into memory.

Additionally, use a context manager to handle the file rather than assigning to a variable, and pass your header to a Session object so you don't have to repeatedly pass header data in your request calls.

Documentation on the PUT method:

https://requests.readthedocs.io/en/master/api/#requests.put

data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the Request.

import requests

session = requests.Session()
session.headers.update(
    {"Authorization": user, "Content-Type": "application/octet-stream"}
)

with open(file_data["name"], "rb") as f:
    response = session.put(url, data=f)

Note: I modified your code to more closely follow python style guides.

Answer 2

EDIT: use io.StringIO() as your file-like object when your writing your dict to csv. Then call get_value() and pass that as your data param to requests.put().

See this question for more details: How do I write data into CSV format as string (not file)?.

Old answer:

If your dict is this:

my_dict = {'col1': 1, 'col2': 2}

then you could convert it to a csv format like so:

csv_data = ','.join(list(my_dict.keys()))
csv_data += ','.join(list(my_dict.values()))
csv_data = csv_data.encode('utf8')

And then do your requests.put() call with data=csv_data.