TypeError: Object of type 'ndarray' is not JSON serializable

Question

knownEmbeddings = []
knownNames = []

for (i, imagePath) in enumerate(imagePaths):
    ## SOME CODE
    knownNames.append(name)
    knownEmbeddings.append(vec.flatten())


data = {"embeddings": knownEmbeddings, "names": knownNames}

f = open('file.json', "wb")
f.write(json.dumps(data, indent=4))
f.close()

Here is the data looks like:

{'embeddings': [array([ 2.23568859e-04, -4.08176295e-02, -1.56606492e-02, -1.40566211e-02,
        5.53448219e-04,  1.34807974e-01,  2.10583732e-02, -7.99260102e-03,
        8.04360434e-02,  2.51036473e-02, -2.45967298e-03,  8.73192959e-03,
        1.08047323e-02,  8.02712217e-02,  6.31465465e-02,  9.41963419e-02],
      dtype=float32), array([-5.54675907e-02,  1.19409459e-02, -3.03599555e-02, -2.86714472e-02,
        6.26528710e-02,  1.25348523e-01, -2.16291733e-02, -4.60545160e-02,
        6.25465512e-02, -7.61162862e-02,  4.28330414e-02,  8.57844874e-02,
        3.75184380e-02, -8.10878351e-02, -8.96525383e-02,  8.15552175e-02,
       -9.75750014e-02, -8.24848488e-02,  9.30746570e-02,  1.71318889e-01,
        1.00642473e-01,  5.39120510e-02,  1.12627009e-02,  1.40678780e-02,
       -4.41719554e-02,  1.03237763e-01,  4.38372791e-02,  7.53327608e-02],
      dtype=float32), array([-0.03736538, -0.0485549 , -0.0976112 , -0.06195155,  0.00269726,
        0.07389018, -0.07325964,  0.06653353, -0.04690087, -0.02606474,
        0.03597135,  0.06904668,  0.02198682, -0.06437466, -0.04554454,
        0.01083081, -0.06293067,  0.07047471,  0.02824293, -0.15522538,
       -0.01900602,  0.10689866, -0.07408814, -0.0419201 ,  0.10475922,
        0.04784475, -0.09085421, -0.20360689,  0.08321641,  0.08441921,
        0.01941148,  0.03566081, -0.05956643,  0.005247  , -0.03989819,
        0.02512971,  0.00458561,  0.13706829], dtype=float32), array([ 7.87236728e-03,  5.65276742e-02, -7.17918649e-02, -1.84332877e-02,
        1.28411269e-02,  2.85971135e-01,  3.13642109e-03,  2.48481780e-02,
       -9.48547944e-02,  2.89725009e-02,  4.33916636e-02,  9.01726633e-02,
        4.36290540e-02, -1.02897413e-01,  2.22285688e-02, -5.19381762e-02,
        1.52556881e-01, -1.25146270e-01,  3.18806712e-03, -2.51053665e-02,
       -4.36606398e-03,  7.19061792e-02,  4.66747172e-02,  8.13280419e-02],
      dtype=float32), array([ 0.09142991, -0.05100765, -0.09615178, -0.03553161,  0.11363017,
        0.19886988,  0.11280693,  0.0229619 , -0.0220201 , -0.01211688,
        0.07489388,  0.0802715 ,  0.16185616, -0.0904082 ,  0.0025941 ,
        0.12167819, -0.07357537, -0.01442344, -0.01343578,  0.16952834,
        0.03366659, -0.0534111 , -0.01595308,  0.15053654, -0.07398864,
        0.04694209, -0.06523879,  0.01342433], dtype=float32), array([-0.05331187,  0.08159426, -0.01742208,  0.00992642, -0.01155609,
        0.25759327, -0.00505029, -0.09290393,  0.01588799, -0.00478396,
        0.08572742, -0.05053008,  0.05197625,  0.1267016 ,  0.15398905,
        0.13668832, -0.13869229,  0.02502107, -0.04443422, -0.05987623,
        0.14948404,  0.03311499,  0.12621029], dtype=float32), array([ 0.1219558 , -0.0371135 , -0.13762642,  0.00431138,  0.20073804,
        0.09986125,  0.21617071,  0.02764285, -0.1352063 ,  0.02268699,
       -0.04734468,  0.10888206,  0.13558514, -0.00319178,  0.02979032,
        0.03558976, -0.07293532, -0.05351996, -0.02449711,  0.1459181 ,
       -0.00320001,  0.01020296, -0.05007216,  0.05868218, -0.03522768,
       -0.01064874, -0.0732395 , -0.05393502], dtype=float32), array([ 0.10833652,  0.08779355, -0.15162815, -0.03925862,  0.08713786,
        0.2850307 ,  0.13499181,  0.01792248, -0.1405847 ,  0.08626581,
        0.02001712, -0.06957201, -0.00727825,  0.01650161,  0.11886367,
        0.07897119, -0.14108546,  0.03840445,  0.05881708,  0.03361814,
       -0.0106756 , -0.04287936, -0.06621028], dtype=float32), array([ 4.90577929e-02,  9.13119391e-02, -2.76884548e-02, -5.19143604e-02,
        1.50506735e-01,  1.86451554e-01,  9.94046330e-02, -7.73873506e-03,
       -1.91362634e-01,  4.69892733e-02, -5.67045361e-02,  2.81608831e-02,
        5.74332848e-02, -9.09122005e-02,  1.46917075e-01,  4.63287433e-04,
        4.22818065e-02, -2.01395284e-02,  1.31114023e-02, -6.61114752e-02],
      dtype=float32), array([ 1.13848910e-01,  1.16239523e-03, -6.73869327e-02,  8.96331621e-05,
        6.71111122e-02,  2.01299891e-01,  1.76381439e-01,  1.44544961e-02,
       -1.36415318e-01, -3.18108648e-02, -3.51585075e-02,  1.24862537e-01,
        6.54390603e-02, -1.79662079e-01,  8.39038659e-03, -6.52492717e-02,
       -4.79320846e-02, -4.05376814e-02, -1.82695538e-02,  1.35992825e-01,
        6.97307214e-02, -5.41270301e-02,  3.14575769e-02,  2.86752880e-02,
        9.04180668e-03,  3.10734902e-02, -3.88299376e-02, -7.43401796e-02],
      dtype=float32), array([ 0.09236415,  0.05246023, -0.03693461,  0.05469636,  0.05779893,
        0.13331857,  0.21085702, -0.01114039, -0.09325632,  0.07158454,
        0.03167493,  0.13376454,  0.13156445, -0.12092946, -0.02573274,
       -0.05352074,  0.00177706,  0.05248505, -0.07331309,  0.06653137,
       -0.02102634,  0.00347302, -0.19828801, -0.08791062,  0.05434143,
        0.07060813, -0.09335811, -0.04778329, -0.02983012,  0.1595401 ,
        0.01018381, -0.04852933, -0.03336967,  0.02886004,  0.05975606,
        0.0974864 , -0.00946077, -0.06796782], dtype=float32), array([ 7.70701468e-02,  2.34568515e-03, -1.22838043e-01,  6.06604481e-05,
        1.08674861e-01,  2.40898028e-01,  7.23511800e-02, -4.14036550e-02,
       -1.20636895e-01, -2.74732499e-03, -1.84729435e-02,  6.18617162e-02,
        8.97722915e-02, -1.62845016e-01, -1.34318219e-02,  5.31670935e-02,
       -8.27090293e-02, -1.22121066e-01,  1.53016988e-02,  1.22807577e-01,
       -1.36648446e-01,  1.32556446e-02, -8.84201974e-02,  8.29895660e-02,
        5.18502928e-02, -7.32250437e-02,  6.30651340e-02, -9.98577196e-03,
       -4.71815281e-02,  6.02727272e-02, -7.98970908e-02, -2.12689787e-02],
      dtype=float32)], 'names': ['adam', 'adam', 'adam', 'adam', 'adam', 'adam', 'bhalla', 'bhalla', 'bhalla', 'bhalla', 'bhalla', 'bhalla']}

I need to save it in json file. When I debug, the data shows as type dict so I used json.dumps(data) to save it in json file but it is throwing below error:

TypeError: Object of type 'ndarray' is not JSON serializable

How can I resolve it

See @S Andrew dictionary key `embeddings` contain `ndarray` object list. — bharatk, Jul 30 '19 at 10:42
before `json.dumps()`, you should convert `ndarray` object into list. — bharatk, Jul 30 '19 at 10:44
@bharatk actaully `knownEmbeddings` is initialized as `knownEmbeddings = []` so there is no option coming for it to convert to list. — S Andrew, Jul 30 '19 at 10:47
`data = {"embeddings": knownEmbeddings.tolist(), "names": knownNames}` , you can retrieve the data to ndarray using `np.asarray(data["embeddings"])` — Shijith, Jul 30 '19 at 10:50
@S Andrew I think you are appending `ndarray` object into `knownEmbeddings` list after initialized, right? — bharatk, Jul 30 '19 at 10:57
@S Andrew Update your question, what are you doing after `initialized` `knownEmbeddings = []`. — bharatk, Jul 30 '19 at 11:02
@S Andrew convert this `vec.flatten()` object into list, before appending into `knownEmbeddings` list. like `knownEmbeddings.append(list(vec.flatten()))` — bharatk, Jul 30 '19 at 11:08
With `knownEmbeddings.append(list(vec.flatten()))`, I am getting `Object of type 'float32' is not JSON serializable` — S Andrew, Jul 30 '19 at 11:11
@S Andrew no more guesses, you should update your question with, what is `vec.flatten()` of `value` and `type` conversion. — bharatk, Jul 30 '19 at 11:14

score 10 · Answer 1 · edited Jul 22 '21 at 11:26

Adding from previous answers and this post, you can do something like this:

class NumpyEncoder(json.JSONEncoder):
    """ Special json encoder for numpy types """
    def default(self, obj):
        if isinstance(obj, (np.int_, np.intc, np.intp, np.int8,
                            np.int16, np.int32, np.int64, np.uint8,
                            np.uint16, np.uint32, np.uint64)):
            return int(obj)
        elif isinstance(obj, (np.float_, np.float16, np.float32,
                              np.float64)):
            return float(obj)
        elif isinstance(obj, (np.ndarray,)):
            return obj.tolist()
        return json.JSONEncoder.default(self, obj)

Then you can call

import numpy as np
import json
operation = json.dumps(data, cls=NumpyEncoder)

In case you are working with a dictionary and want to save the modified dictionary but receiving a similar error

'TypeError: Object of type intc is not JSON serializable'

you can use

dumped = json.dumps(data, cls=NumpyEncoder)
with open('json_file_path.json', 'a') as f:
    f.write(dumped + '\n')

The code ensure the save json file does not contain \ splashes as stated in here

I was dreading having to write my own encoder! Your code worked perfectly! — abrac, May 25 '21 at 06:57
Great work, works on my dictionary that has some arrays in it, changing them to a list. You can also change `return obj.tolist()` to `str(obj)` if you rather want to see the mere string of an array instead of a list of an array. This can make sense if you have larger arrays which will take far too much printing space when each list element gets a new line. — questionto42, Jul 22 '21 at 12:14

Sam Mason · Answer 2 · 2021-07-22T10:17:07.837

2

you want to convert to a list of 64bit floats first, e.g.

import json
import numpy as np

arr = np.array([1], dtype=np.float32)

json.dumps(list(arr))

gives me: TypeError: Object of type float32 is not JSON serializable

while if I convert to 64 bit floating point values before serialising to JSON, it works:

json.dumps(list(arr.astype(float)))

I successfully get '[1.0]' back

also note that if you have a file object, you're much better using json.dump instead, e.g. try doing:

with open('file.json', "w") as f:
    json.dump(data, f, indent=4)

edited Jul 22 '21 at 10:17

answered Jul 30 '19 at 11:25

Sam Mason

15,216
1
41
60

1

This with arr.astype(float).tolist() instead of list(arr.astype(float)) works for me. – sigma1510 Aug 04 '22 at 08:41
as @JaredH points out you can even just use [`arr.tolist()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.tolist.html) without the call to `astype` as this will automatically convert to numpy scalars into Python scalars (i.e. a `np.float32` into a `float`) – Sam Mason Aug 08 '22 at 11:50

score 2 · Answer 3 · answered Jul 07 '20 at 17:05

2

If you want to convert to a list to follow bharatk's solution above, I recommend using the Numpy tolist() method.

data = {"embeddings": knownEmbeddings.tolist(), "names": knownNames.tolist()}

Good luck!

answered Jul 07 '20 at 17:05

Jared H

21
2

score 1 · Answer 4 · answered Jul 30 '19 at 11:04

data is a dict indeed, but it contains lists of ndarrays, which need to be serialized too, and are not (natively) serializable.

What you need here is to provide your own JSONEncoder that knows how to deal with ndarray (turning them into lists as suggested by bharatk being the most obvious solution).

TypeError: Object of type 'ndarray' is not JSON serializable

4 Answers4

Linked