Failing to convert numpy array to JSON

Question

I am trying to write a JSONEncoder so I can dump some of my Python object data to JSON text to be read in by another application. I am having particular trouble converting the numpy arrays. I have read many of the SO articles on the subject, and I have borrowed heavily from @tlausch's solution in this article SimpleJSON and NumPy array - but I have not been able to get it working.

I have a pretty simple class that demonstrates my problem:

import numpy as np
class Movement:
    def __init__(self, t1,t2,t3,t4):
        self._t1 = t1
        self._t2 = t2
        self._t3 = t3
        self._t4 = t4
        self._a0 = np.array([0,0,0])

Here is the code from my JSONEncoder:

import json
import base64
import numpy as np
class MovementEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.ndarray):
            if obj.flags['C_CONTIGUOUS']:
                obj_data = obj.data
            else:
                cont_obj = np.ascontiguousarray(obj)
                assert(cont_obj.flags['C_CONTIGUOUS'])
                obj_data = cont_obj.data
            data_b64 = base64.b64encode(obj_data)
            return dict(__ndarray__ = data_b64, dtype = str(obj.dtype), shape = obj.shape)
        try:
            my_dict = obj.__dict__   ## <-- ERROR raised here
        except TypeError:
            pass
        else:
            return my_dict
        return json.JSONEncoder.default(self, obj)

When I create a simple Movement object, and then call encoder.encode(obj), I get the following error:

>>> obj = Movement(1,2,3,4)
>>> encoder = MovementEncoder()
>>> encoder.encode(obj)
...
    'bytes' object has no attribute '__dict__'

By adding some print statements to the encoder, I can see that it is correctly recursing from the object, to the object's dictionary, then to the attribute that has the np.array type. I thought the point of this solution was that the base64 representation of the ndarray type was JSON encodable by default, but that appears not to be the case. Where did I go wrong?

Note: using Python 3.4, and NumPy 1.8.2

EDIT: updated code to show where error occurs

Your code runs successfully for me on python 2.7.5 with numpy 1.7.1 — ForeverWintr, Apr 05 '16 at 19:53
Based on the full traceback, which line is throwing that error? — Alex Hall, Apr 05 '16 at 20:25
@AlexHall: If I catch AttributeError and pass through to call the json.JSONEncoder.default() method, then I get a TypeError from that call, saying that `b'AAAAAAAAAAAAAAA'` is not JSON serializable — gariepy, Apr 05 '16 at 20:44
Good, that's progress. For the next error: You'd get the same if you just wrote `json.dumps(b'123')`. That's because in Python 3 raw byte strings can't be JSON encoded. You have to convert to a unicode object first, thus forcing you to pick a character encoding. — Alex Hall, Apr 05 '16 at 20:59

score 3 · Accepted Answer · answered Apr 22 '16 at 19:20

I was finally able to solve this problem by adjusting the return value for numpy ndarray types to return a list, instead of a dict. I used the numpy builtin method tolist(), which returns a json-encodable representation of an ndarray.

import json
import base64
import numpy as np
class MovementEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.ndarray):
            if obj.flags['C_CONTIGUOUS']:
                obj_data = obj.data
            else:
                cont_obj = np.ascontiguousarray(obj)
                assert(cont_obj.flags['C_CONTIGUOUS'])
                obj_data = cont_obj.data
            ## data_b64 = base64.b64encode(obj_data)
            ## converting to base64 and returning a dictionary did not work
            ## return dict(__ndarray__ = data_b64, dtype = str(obj.dtype), shape = obj.shape)
            return obj.tolist()  ## instead, utilize numpy builtin tolist() method
        try:
            my_dict = obj.__dict__   ## <-- ERROR raised here
        except TypeError:
            pass
        else:
            return my_dict
        return json.JSONEncoder.default(self, obj)

Failing to convert numpy array to JSON

1 Answers1