1

I'm using pythons Requests to send a Pandas Dataframe to a Flask server. The dataframe has about 2 million rows and 16 columns. I want to send a config dictionary along with the dataframe, as metadata. At the moment I am able to send the dataframe as a JSON file, however, I can't find any way to attach the metadata in the same post request.

Here's my code:

Client side:

 # Post request containing 1. The dataset (pandas df) 2. The metadata (dict)

 dataset = dataset.to_json(orient='split')
 metadata = {'dataset ID': "makis", 'date start': "1", 'date end': "2"}

 url = "http://localhost:8081/upload_dataset"
 r = requests.post(url, data=dataset)
 return r.text

Server side:

@app.route("/upload_dataset", methods=['POST'])

def upload_dataset():
    from werkzeug.datastructures import FileStorage

    payload = request.stream
    dataset = pd.read_json(payload, typ='frame', orient='split')
    FileStorage(payload).save('dataset.csv')

    return 'File Uploaded & Standing by', 200
Nazim Kerimbekov
  • 4,712
  • 8
  • 34
  • 58
Meeron
  • 13
  • 1
  • 3

1 Answers1

1

Once serialized to json, your dataset is plain text. To add more parameters from there, your only options are embed your payload along with metadata in post parameters, resulting in url-encoding the json. Or embed your payload in a top-level json post, thus double-encode in json.

You would gain in clarity and maybe performance if you left json encoding job to requests instead. In this way you could add data and still encode/decode only once.

Example


dataset = dataset.to_dict(orient='list')
post_data = {'dataset ID': "makis", 'date start': "1", 'date end': "2", 'payload': dataset}
url = "http://localhost:8081/upload_dataset"
r = requests.post(url, json=post_data)

Server side:

@app.route("/upload_dataset", methods=['POST'])
def upload_dataset():
    post_data = request.get_json()
    ## Use of meta data keys e.g. post_data['date start']
    dataset = pd.from_dict(post_data['payload'], orient='columns')
Diane M
  • 1,503
  • 1
  • 12
  • 23
  • 1
    yup, that works just fine, thanks :). Minor correction: at the server side the first line is dataset = pd.DataFrame.from_dict(post_data.get('dataset'), orient='columns'), whereas at the client it's dataset = dataset.to_dict(orient='list'). At least that's how i got it to work. Thanks! – Meeron May 08 '19 at 13:20
  • Sorry, I didnt test and assumed to_dict and to_json shared the same API, which they seem not. I made the edits – Diane M May 08 '19 at 16:08