Flask and BytesIO: Problem to output file as a stream

Question

I'm trying to output generated Excel file as a filestream for JSON payload as follow:


    def post(self):

        keys = []
        wb = Workbook()
        ws = wb.active

        json_data = request.get_json()


        with NamedTemporaryFile() as tmp:
            for i in range(len(json_data)):
                sub_obj = json_data[i]
                if i == 0:
                    keys = list(sub_obj.keys())
                    for k in range(len(keys)):
                        # row or column index start from 1
                        ws.cell(row=(i + 1), column=(k + 1), value=keys[k]);
                for j in range(len(keys)):
                    ws.cell(row=(i + 2), column=(j + 1), value=sub_obj[keys[j]]);

            wb.save(tmp.name)
            output = BytesIO(tmp.read())
            return output.getvalue()

but that will throw an error

Traceback (most recent call last):
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask_restful/__init__.py", line 272, in error_router
    return original_handler(e)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/_compat.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask_restful/__init__.py", line 272, in error_router
    return original_handler(e)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/_compat.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask_restful/__init__.py", line 472, in wrapper
    return self.make_response(data, code, headers=headers)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask_restful/__init__.py", line 501, in make_response
    resp = self.representations[mediatype](data, *args, **kwargs)
  File "/PythonAPI/flask/env/lib/python3.7/site-packages/flask_restful/representations/json.py", line 21, in output_json
    dumped = dumps(data, **settings) + "\n"
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

when

    def post(self):

        keys = []
        wb = Workbook()
        ws = wb.active

        json_data = request.get_json()

        for i in range(len(json_data)):
            sub_obj = json_data[i]
            if i == 0:
                keys = list(sub_obj.keys())
                for k in range(len(keys)):
                    # row or column index start from 1
                    ws.cell(row=(i + 1), column=(k + 1), value=keys[k]);
            for j in range(len(keys)):
                ws.cell(row=(i + 2), column=(j + 1), value=sub_obj[keys[j]]);
        wb.save("test.xlsx")

works fine.

Any thoughts what happen to the JSNO data in the first example?

PS. I think it must to be something related to the Flask API response as default response is JSON but overwriting it with

return output.getvalue(), 200, {'content-type': 'application/octet-stream'}

Is not working either.

When posting a question about code that produces an Exception, always include the complete Traceback - copy and paste it then format it as code (select it and type `ctrl-k`) - I don't see any attempt to serialize (json.dump(s)) in your code unless it is happening with the request method in which case you haven't provided a [mre]. — wwii, Jun 22 '20 at 20:13
@wii OP updated with the full code set for defined function for that part of the code — JackTheKnife, Jun 22 '20 at 20:20
Did you try commenting out each of the last three lines in turn to see which is causing the problem - it isn't evident from the Traceback. Then after finding the problematic line did you inspect the data being operated on? Did you consider either of these to fix that data? - [set object is not JSON serializable](https://stackoverflow.com/questions/22281059/set-object-is-not-json-serializable), [https://stackoverflow.com/questions/8230315/how-to-json-serialize-sets](https://stackoverflow.com/questions/8230315/how-to-json-serialize-sets). — wwii, Jun 22 '20 at 20:32
@wwii I just did what you have suggest - commented out lines and it looks like it may be Flask response back - type JSON rather than stream — JackTheKnife, Jun 22 '20 at 20:34
Looks like the Traceback message has changed - those link are not relevant. — wwii, Jun 22 '20 at 20:36
Possibly related? [Object of type 'bytes' is not JSON serializable when upgrading my python environment](https://stackoverflow.com/questions/44605393/object-of-type-bytes-is-not-json-serializable-when-upgrading-my-python-environ) — wwii, Jun 22 '20 at 20:42

score 1 · Accepted Answer · answered Jun 22 '20 at 22:21

Got it solved by changing default Flask response type header as follow:

    keys = []
    wb = Workbook()
    ws = wb.active

    json_data = request.get_json()

    with NamedTemporaryFile() as tmp:

        for i in range(len(json_data)):
            sub_obj = json_data[i]
            if i == 0:
                keys = list(sub_obj.keys())
                for k in range(len(keys)):
                    # row or column index start from 1
                    ws.cell(row=(i + 1), column=(k + 1), value=keys[k]);
            for j in range(len(keys)):
                ws.cell(row=(i + 2), column=(j + 1), value=sub_obj[keys[j]]);
        wb.save(tmp.name)

        buf = BytesIO(tmp.read())

        response = app.make_response(buf.getvalue())
        response.headers['content-type'] = 'application/octet-stream'

        return response

Flask and BytesIO: Problem to output file as a stream

1 Answers1