8

A numpy array should be sent from python to JS through flask. I don't want to jsonify it and send as it would increase the size of the response and eventually the response time. So I thought I could convert numpy into bytes using tobytes(), send the bytes to JS through flask and convert the bytes back to float in JS and found Converting a string of packed bytes into an array of floats in Javascript answer helpful.

The list of operations are

byte_arr = np.array([5.6], dtype=np.float32).tobytes()  # Used one value just to make it simple
return byte_arr  # Bytes can be directly sent from flask as response

In JS,

str = response.text
bytes = Uint8Array.from(str, c => c.charCodeAt(0))
floats = new Float32Array(bytes.buffer)

But when converting to float in JS, I am not getting the correct values. When debugging I found that in JS some values in Uint8 array are not matching with the byte values that are sent from python. 

JS: console.log(bytes)  // [51, 51, 255, 64]
Python: for val in byte_arr: print(val, end=" ")  // 51, 51, 179, 64

Just to verify, I created another client in python, made the same request and converted the response bytes to float. I encountered the same issue here also. So I narrowed down that the problem is on the server side not on the client side. One more interesting thing I found when playing with multiple values are, only the numbers which are greater than 127 (I guess so) are getting converted to 255.

While trying some random hacks, I tried to convert the byte integer to character using chr() method of python before sending the response and the conversion on the client side worked.

byte_arr = np.array([5.6], dtype=np.float32).tobytes()
byte_arr_char = "".join([chr(i) for i in byte_arr])
return byte_arr_char

But the question I have is, Is this an ideal solution or am I doing some hack here to make it work? Can anyone help me to understand why sending plain bytes without chr() doesn't work?

Suba Selvandran
  • 304
  • 2
  • 16
  • Can you edit into the question what the value of `byte_arr` is in python, and what the value of `response.text` is in JS? – mostsquares Mar 01 '19 at 21:39
  • How is the `response` received? Are you making sure to [set a suitable response type](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Sending_and_Receiving_Binary_Data)? – Martijn Pieters Mar 03 '19 at 14:50
  • @CharlieWindolf Sure. But the value in JS script is all question marks. – Suba Selvandran Mar 06 '19 at 09:11

1 Answers1

11

If you don't set the MIME type explicitly, I think Flask will treat it as text data. Your browser seems to have decoded the binary data with ASCII, which can explain why only values larger than 127 were affected.

Therefore, please try setting the Content-Type of the response in Flask:

@app.route('/your/url/to/numpy/data')
def get_nparray():
    your_np_array = np.array([5.6], dtype=np.float32)
    response = flask.make_response(your_np_array.tobytes())
    response.headers.set('Content-Type', 'application/octet-stream')
    # response.headers.set('Content-Disposition', 'attachment', filename='np-array.bin')
    return response

Alternatively, there is a helper function flask.send_file to construct the response in a single line. Please find an example here.

Besides this error, also pay attention to the endianness of your binary data, which is hardware specific. I would refer you to this answer (Javascript Typed Arrays and Endianness).

gdlmx
  • 6,479
  • 1
  • 21
  • 39
  • Thanks for the answer. It works the way I wanted. Also, I read about the usage of send_file but the data will be received as an image, in that case, I wasn't sure how to receive the image as an array/text. – Suba Selvandran Mar 06 '19 at 09:22
  • I actually encountered another issue here. I am able to get the correct response in python but in JS where I am using angular's $http to make the response, I am still stuck with the same issue. The response I get is not a "application/octet-stream" response. This summarizes the problem https://github.com/angular/angular.js/issues/10349. Any solution for this? – Suba Selvandran Mar 07 '19 at 11:27