1

For my data (should be some GB) that looks like

data = d1 = {'1': {'1': datetime.datetime.now()}, '2': {'2': datetime.datetime.now(), '3': datetime.datetime.now()}, '3': set()}

I wrote a custom JSON encoder

class _JSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime.datetime):
            return obj.isoformat()
        elif isinstance(obj, set):
            return {o: None for o in obj} 
        else:
            return json.JSONEncoder.default(self, obj)

Although

s = json.dumps(data, cls=_JSONEncoder)

works fine for smaller data, but it does not converge/end for larger dicts. What am I doing wrong? It works with orjson though. Is the standard implementation so slow?

Michael Dorner
  • 17,587
  • 13
  • 87
  • 117
  • 1
    I suppose building "some GB" of JSON in-memory could be an issue. Streaming encoders might work better. This sounds interesting: https://stackoverflow.com/questions/12670395/json-encoding-very-long-iterators – Tomalak Sep 20 '21 at 11:08
  • Thanks and I thought it is just me being stupid. :) – Michael Dorner Sep 20 '21 at 11:26
  • Well I'm not speaking authoritatively, that's just my gut feeling. I would not be surprised if the standard `json` module is not optimized for these kinds of workloads. – Tomalak Sep 20 '21 at 11:39

0 Answers0