1

According to Consumer Surveys docs, the questions[].images[].data field takes a bytes datatype.

I'm using Python 3 for implementation, but the API is giving errors like Invalid ByteString or bytes type is not JSON serializable.

I'm using the following code:

    import base64
    import urllib
    url = 'http://example.com/image.png'
    raw_img = urllib.request.urlopen(url).read()

    # is not JSON serializable due to json serializer not being able to serialize raw bytes
    img_data = raw_img

    # next errors: Invalid ByteString, when tried with base64 encoding as followings:
    img_data = base64.b64encode(raw_img)
    # Also tried decoding it to UTF.8 `.decode('utf-8')`

img_data is part of the JSON payload that is being sent to the API.

Am I missing something? what's the correct way to handle image data upload for questions? I looked into https://github.com/google/consumer-surveys/tree/master/python/src but there is not example of this part.

Thanks

A Anez
  • 23
  • 4
  • Possible duplicate of [What is the correct format for the API SurveyQuestionImage.Data field?](http://stackoverflow.com/questions/38575967/what-is-the-correct-format-for-the-api-surveyquestionimage-data-field) – Maia Werbos Aug 16 '16 at 17:02
  • @MaiaWerbos Sure, I just didn't have enough rep to comment there and still had the issue, so I thought about asking myself. – A Anez Aug 16 '16 at 21:21

1 Answers1

2

You need to use web-safe/URL-safe encoding. Here's some documentation on doing this in Python: https://pymotw.com/2/base64/#url-safe-variations

In your case, this would look like

img_data = base64.urlsafe_b64encode(raw_img)

ETA: In Python 3, the API expects the image data to be of type str so it can be JSON serialized, but the base64.urlsafe_b64encode method returns the data in the form of UTF-8 bytes. You can fix this by converting the bytes to Unicode:

img_data = base64.urlsafe_b64encode(raw_img)
img_data = img_data.decode('utf-8')
Maia Werbos
  • 164
  • 9
  • Thanks Maia, but problem persists, I had tried that also: googleapiclient.errors.HttpError: – A Anez Aug 16 '16 at 21:19
  • It actually throws a `is not JSON serializable` error. After decoding it to utf-8 GCS API just throws a Backend error: without RequestId or further information – A Anez Aug 16 '16 at 21:33
  • I am using python 3, forgot to mention it in the question. – A Anez Aug 16 '16 at 21:41
  • Can you make your request without the "?alt=json" parameter? – Maia Werbos Aug 18 '16 at 00:42
  • I had to modify `googleapiclient` pypi package to supress the alt param for this test, but it still gives a 500 Backend Error. I also tried quoting the data values: https://docs.python.org/3/library/urllib.parse.html – A Anez Aug 19 '16 at 16:48
  • I'm unable to reproduce your error, using this exact base64 library function. I'm using code like that here: https://github.com/google/consumer-surveys/blob/master/python/src/example_client.py#L284 and doing: `img_data = base64.urlsafe_b64encode(raw_img)` and setting `'data': img_data` in the body_def of the survey. I can create a survey this way with no issues, so I'm not sure what the problem is. The 400 errors at least should be giving you a request_id; if you post that I can give you more info. – Maia Werbos Aug 19 '16 at 23:38
  • I pastied my implementation here: http://pastebin.com/9mY3HfYt It is running on a python 3 virtualenv. The error comes from the googleapiclient trying to serialize the urlsafe encoded image. Did you test it on python 3? you should be able tor reproduce the error on your side – A Anez Aug 21 '16 at 16:08
  • Ah, I see. I tested on Python 3 and saw `raise TypeError(repr(o) + " is not JSON serializable")`. It looks like the python3 version of the `base64` library returns a `bytes` from this method rather than a `str`. I fixed this problem by adding the line: `img_data = img_data.decode('utf-8')` to convert to str. – Maia Werbos Aug 22 '16 at 21:29
  • I had tried that before, that's exactly wher the 500 Backend error happens: 'data': base64.urlsafe_b64encode(raw_img).decode('utf-8'). Unfortunately it does not give a RequestId for you to trace – A Anez Aug 23 '16 at 23:13
  • turns out the error (HTTP 500) comes from a survey with multiple questions of the type multipleAnswersWithImage. My original request, which I'm using right now, contained more than one of those questions types . So I was correctly sending the data, but didn't have the lead to track the error there. – A Anez Aug 23 '16 at 23:35
  • That sounds like a bug on our end. Sorry about that! We should never be returning 500 errors in that situation. Would you be willing to email the details of your failed request to gcs-api@google.com so we can investigate? We really appreciate reports like this. – Maia Werbos Aug 24 '16 at 00:20
  • Thanks Maia, but I was told that address is for the sales team. So perhaps I can open an issue on the github repo or just leave these comments here? – A Anez Aug 25 '16 at 15:22
  • Hi Arnaldo - I'm also on the GCS team and am filling in for Maia. Can you leave comments here? In the meantime I'll check into the gcs-api list. Thank you! – Tiem Song Aug 30 '16 at 21:18