0

To save some sensor data on a RPI in CouchDB using pycouchdb, I created a database model class to have a clear structure instead of loosely typed dicts

class SensorMeasure(NamedTuple):
  temp: float
  soilMoisture: float
  dateTime: datetime

Since it seems not possible to serialize this object automatically, I used the _asdict() method from NamedTuple to get an dict object, which could be stored in the database

server = pycouchdb.Server("http://127.0.0.1:5984/")
db = server.database(dbName)
measure = SensorMeasure(temp=sensor.getTemperature(), soilMoisture = sensor.getMoisture(), dateTime = datetime.now())
db.save(measure._asdict())

While this works well for primitive types like float, it breaks on datetime:

TypeError: Object of type datetime is not JSON serializable

It seems like that I have to tell the serializer how he could generate a string from the datetime object, which seems not possible for me without modifing pycouchdbs source code.

The only working workaround seems using string instead of datetime in the SensorMeasure model and use the isoformat() method of datetime. But this would require me to

  1. install additional libraries for parsing
  2. I have to parse it on every usage with the overhead of creating a new object, specify the format, ...

In terms of design, it would be much better to have a datetime attribute in the class. How can I archive this?

Other workaround thoughts

With the zip function it seems possible to define which keys should be serialized. This leads me to the idea of removing the dateTime field and then re-adding it as string value like this:

class SensorMeasure(NamedTuple):
  temp:float
  soilMoisture: float
  dateTime: datetime

  def test(self):
    serializeFields = list(self._fields)
    del serializeFields['dateTime']

    serialized = OrderedDict(zip(serializeFields, self))
    print(serialized)

    serialized['dateTime'] = dateTime.isoformat()
    print(serialized)

But this doesn't work since the returned tuple is immutable. Converting it to a list should allow writing, however the lists seems only allowing integer keys:

TypeError: list indices must be integers or slices, not str
Lion
  • 16,606
  • 23
  • 86
  • 148

2 Answers2

0

Suboptimal workaround

This is just for documentation purpose/completeness. Scroll down for a better solution

With a few converting operations, the workaround idea from my question works:

class SensorMeasure(NamedTuple):
  temp: float
  soilMoisture: float
  dateTime: datetime

  def test(self):
    serializeFields = list(self._fields)
    del serializeFields[serializeFields.index('dateTime')]

    serialized = OrderedDict(zip(serializeFields, self))
    tmp = list(serialized.items())
    tmp.append(('dateTime', self.dateTime.isoformat()))

    finalDict = OrderedDict(tmp)
    return finalDict

But it is not only a lot of code/work for datetime serialization, which is required in every model. Or at least a call in every model to some helper class.

Better solution: Custom db.save method

A better solution would be setting default parsing method on json.dumps

jsonStr = json.dumps(doc, default = str)

Sadly pycouchdb currently doesn't allow this. Of course I can send pull request that extend this functionality by e.g. a new parameter. But for now, I just create my own save method as a quick and suiteable fix.

def save(sensorMeasure):
  doc = copy.deepcopy(sensorMeasure._asdict())
  doc['_id'] = uuid.uuid4().hex

  # Default is important to parse datetime objects
  jsonStr = json.dumps(doc, default = str)
  data = pycouchdb.utils.force_bytes(jsonStr)
  (resp, results) = db.resource(doc['_id']).put(data=data)

The only problem being still present is the lack of typing when using db.get():

doc = db.get("bb29e02dc2364a57ac1e707d7dc2134b")

This returns an dictionary, so parsing the string to a datetime object is still required. It could be done using a constructor. Not as easy as I know it e.g. from ASP.NET Core, where serialization/deserialization could be done with a single line of code, but seems possible.

Lion
  • 16,606
  • 23
  • 86
  • 148
0

It is possible to define your own decoder/encoder. I did this, works fine for saving:

import json
import couchdb
from datetime import datetime, date

class JSONEncoderExtendDate(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        elif isinstance(obj, date):
            return obj.isoformat()


        return json.JSONEncoder.default(self, obj)

def encode_json(obj):
    return json.dumps(obj, cls=JSONEncoderExtendDate)
def decode_json(obj):
    return json.loads(obj)


couchdb.json.use(decode=decode_json, encode=encode_json)

And then just do your save normally.

Probably should have also implemented the decoder, but that is pretty obvious. I only use saving at the python side. You need to define both the decoder and encoder, that is why it's not empty.

RiverWalker
  • 21
  • 2