Tornado, Motor with mongomock for testing

Question

I'm writing a test module for a tornado based web application. The application uses motor as mongodb connector and I wish that my tests run on a temporary database. I am using a mocking technic on the delegate_class of the connector client as follows:

import json
import mock
import motor
import tornado.ioloop
import tornado.testing
import mongomock
import myapp


patch_motor_client = mock.patch('motor.motor_tornado.MotorClient.__delegate_class__', new=mongomock.MongoClient)
patch_motor_database = mock.patch('motor.motor_tornado.MotorDatabase.__delegate_class__', new=mock.MagicMock)

patch_motor_client.start()
patch_motor_database.start()


class TestHandlerBase(tornado.testing.AsyncHTTPTestCase):
    """
    Base test handler
    """

    def setUp(self):
        # Create your Application for testing
        self.application = myapp.app.Application()
        super(TestHandlerBase, self).setUp()

    def get_app(self):
        return self.application

    def get_new_ioloop(self):
        return tornado.ioloop.IOLoop.instance()


class TestMyHandler(TestHandlerBase):
    def test_post_ok(self):
        """
        POST a resource is OK
        """
        post_args = {
            'data': 'some data here..'
        }

        response = self.fetch('myapi/v1/scripts', method='POST', body=json.dumps(post_args))

        # assert status is 201
        self.assertEqual(response.code, 201)

When I launch my tests, I'm getting this error:

  File "/data/.virtualenvs/myapp/lib/python3.5/site-packages/motor/core.py", line 162, in __getitem__
    return db_class(self, name)
  File "/data/.virtualenvs/myapp/lib/python3.5/site-packages/motor/core.py", line 217, in __init__
    client.delegate, name, **kwargs)
  File "/data/.virtualenvs/myapp/lib/python3.5/site-packages/pymongo/database.py", line 102, in __init__
    read_concern or client.read_concern)
  File "/data/.virtualenvs/myapp/lib/python3.5/site-packages/pymongo/common.py", line 614, in __init__
    raise TypeError("codec_options must be an instance of "
TypeError: codec_options must be an instance of bson.codec_options.CodecOptions

For the moment I'm not able to make it work, and I'm wondering if what I want to do is even possible with the current versions of motor (1.2.1), mongomock (3.8.0) and tornado (4.5.3), or I miss something?

Thanks for all your suggestions.

Answer 1

I could only get this to work with heavy monkey patching (I guess mock.patch-ing would be similar but I was not interested in reverting the changes).

I identified the following problems:

1. Motor seems to ignore __delegate_class__ at times, and instanciate actual pymongo Database or Collection instead (ex delegate = _delegate or Collection(database.delegate, name))
2. Motor wraps the delegate class and wants every single expected attribute to exist, so that it can "asynchronize" them.
3. The motor "agnostic" cursors rely on semi-private details of the pymongo Cursors, like _refresh or __data, because they need to intervene in their manipulations (asynchronize them again, since they're IO). Mongomock cursors are much simpler and do not have such attributes

And worked around them like this:

1. Is probably fixable in motor, but it gave me trouble, so I had to start "motor-wrapping" at the Collection level only (it did not work from the Client or Database), and to hack so that the mongomock database instantiates motor-wrapped collections:

db = mongomock.Database(mongomock.MongoClient(), 'db_name')

# Monkeypatch get_collection so that collections are motor-wrapped
def create_motor_wrapped_mock_collection(
        name, codec_options=None, read_preference=None,
        write_concern=None, read_concern=None):
    if read_concern:
        raise NotImplementedError('Mongomock does not handle read_concern yet')
    collection = db._collections.get(name)
    if collection is None:
        delegate = mongomock.Collection(db, name, write_concern=write_concern)

        # wont be used, as we patch get_io_loop, but the MotorCollection ctor checks type
        fake_client = motor.motor_tornado.MotorClient()
        fake_db = motor.motor_tornado.MotorDatabase(fake_client, 'db_name')

        motor_collection = motor.motor_tornado.MotorCollection(fake_db, name, _delegate=delegate)
        collection = db._collections[name] = motor_collection
        collection.get_io_loop = lambda: tornado.ioloop.IOLoop.current()
    return collection

db.get_collection = create_motor_wrapped_mock_collection 

# Then use db in your code or patch it in

2. Is the problem you're hitting. This is avoidable by scanning the required attributes, and defining fake ones when missing:

def _prepare_for_motor_wrapping(cls, wrapper_cls):

    # Motor expects all attributes to exist on a delegate, to generate wrapped methods/attributes, even the ones we
    # won't need. This patches in dummy attributes/methods so that Motor wrapping can succeed

    def gen_fake_method(name, on):
        def fake_method(*args, **kwargs):
            raise NotImplementedError(name + ' on ' + on)
        return fake_method

    attrs = list(wrapper_cls.__dict__.items()) + list(motor.core.AgnosticBaseProperties.__dict__.items())
    for k, v in attrs:
        attr_name = getattr(v, 'attr_name', None) or k
        if not hasattr(cls, attr_name) and isinstance(v, motor.metaprogramming.MotorAttributeFactory):
            if isinstance(v, motor.metaprogramming.ReadOnlyProperty):
                setattr(cls, attr_name, None)
            elif isinstance(v, motor.metaprogramming.Async) or isinstance(v, motor.metaprogramming.Unwrap):
                setattr(cls, attr_name, gen_fake_method(attr_name, cls.__name__))
            else:
                raise RuntimeError('Dont know how to fake %s' % v)

# We must clear the cache, as classes might have been generated already during some previous import
motor.metaprogramming._class_cache = {}

_prepare_for_motor_wrapping(mongomock.Database, motor.core.AgnosticDatabase)
motor.motor_tornado.MotorDatabase = motor.motor_tornado.create_motor_class(motor.core.AgnosticDatabase)

_prepare_for_motor_wrapping(mongomock.Collection, motor.core.AgnosticCollection) 
motor.motor_tornado.MotorCollection = motor.motor_tornado.create_motor_class(motor.core.AgnosticCollection)

For some reason, MotorClient must be left intact.

3. I got out of by patching to_list(), since that's the only thing I use on the results of coll.aggregate() and coll.find()

def _patch_aggregate_cursor():

    def curs_to_docs(docs_future, curs_future):
        curs = curs_future.result()
        docs_future.set_result(list(curs))

    def to_list(self, *args):
        mock_cursor_future = self.collection._async_aggregate(self.pipeline)
        docs_future = self._framework.get_future(self.get_io_loop())
        self._framework.add_future(
            self.get_io_loop(),
            mock_cursor_future,
            curs_to_docs, docs_future)
        return docs_future

    motor.core.AgnosticAggregationCursor.to_list = to_list


def _patch_generic_cursor():

    def to_list(self, *args):
        docs = list(self.delegate)
        docs_future = self._framework.get_future(self.get_io_loop())
        docs_future.set_result(docs)
        return docs_future

    motor.core.AgnosticCursor.to_list = to_list

All this is probably incomplete and fragile, so I let you judge if it is worth the efforts.