29

I can't figure out what kind of object a sqlalchemy query returns.

entries = session.query(Foo.id, Foo.date).all()

The type of each object in entries seems to be sqlalchemy.util._collections.result, but a quick from sqlalchemy.util._collections import result in a python interpreter raises an ImportError.

What I'm ultimately trying to do is to type hint this function:

def my_super_function(session: Session) -> ???:
    entries = session.query(Foo.id, Foo.date).all()
    return entries

What should I put in place of ???? mypy (in this case) seems to be fine with List[Tuple[int, str]] because yes indeed I can access my entries like if they were tuples, but i can also access them with entry.date, for example.

JPFrancoia
  • 4,866
  • 10
  • 43
  • 73

2 Answers2

33

I too found it curious that the class couldn't be imported. The answer is pretty long as I've walked you through how I've worked it out, bear with me.

Query.all() calls list() on the Query object itself:

def all(self):
    """Return the results represented by this ``Query`` as a list.
    This results in an execution of the underlying query.
    """
    return list(self)

... where list will be iterating over the object, so Query.__iter__():

def __iter__(self):
    context = self._compile_context()
    context.statement.use_labels = True
    if self._autoflush and not self._populate_existing:
        self.session._autoflush()
    return self._execute_and_instances(context)

... returns the result of Query._execute_and_instances() method:

def _execute_and_instances(self, querycontext):
    conn = self._get_bind_args(
        querycontext, self._connection_from_session, close_with_result=True
    )

    result = conn.execute(querycontext.statement, self._params)
    return loading.instances(querycontext.query, result, querycontext)

Which executes the query and returns the result of sqlalchemy.loading.instances() function. In that function there is this line which applies to non-single-entity queries:

keyed_tuple = util.lightweight_named_tuple("result", labels)

... and if I stick a print(keyed_tuple) in after that line it prints <class 'sqlalchemy.util._collections.result'>, which is the type that you mention above. So whatever that object is, it's coming from the sqlalchemy.util._collections.lightweight_named_tuple() function:

def lightweight_named_tuple(name, fields):
    hash_ = (name,) + tuple(fields)
    tp_cls = _lw_tuples.get(hash_)
    if tp_cls:
        return tp_cls

    tp_cls = type(
        name,
        (_LW,),
        dict(
            [
                (field, _property_getters[idx])
                for idx, field in enumerate(fields)
                if field is not None
            ]
            + [("__slots__", ())]
        ),
    )

    tp_cls._real_fields = fields
    tp_cls._fields = tuple([f for f in fields if f is not None])

    _lw_tuples[hash_] = tp_cls
    return tp_cls

So the key part is this statement:

tp_cls = type(
    name,
    (_LW,),
    dict(
        [
            (field, _property_getters[idx])
            for idx, field in enumerate(fields)
            if field is not None
        ]
        + [("__slots__", ())]
    ),
)

... which calls the built in type() class which according to the docs:

With three arguments, return a new type object. This is essentially a dynamic form of the class statement.

And this is why you cannot import the class sqlalchemy.util._collections.result - because the class is only constructed at query time. I'd say that the reason for this is that the column names (i.e. the named tuple attributes) aren't known until the query is executed).

From python docs the signature for type is: type(name, bases, dict) where:

The name string is the class name and becomes the __name__ attribute; the bases tuple itemizes the base classes and becomes the __bases__ attribute; and the dict dictionary is the namespace containing definitions for class body and is copied to a standard dictionary to become the __dict__ attribute.

As you can see, the bases argument passed to type() in lightweight_named_tuple() is (_LW,). So any of the dynamically created named tuple types inherit from sqlalchemy.util._collections._LW, which is a class that you can import:

from sqlalchemy.util._collections import _LW

entries = session.query(Foo.id, Foo.date).all()
for entry in entries:
    assert isinstance(entry, _LW)  # True

... so I'm not sure whether it's good form to type your function to an internal class with the leading underscore, but _LW inherits from sqlalchemy.util._collections.AbstractKeyedTuple, which itself inherits from tuple. That's why your current typing of List[Tuple[int, str]] works, because it is a list of tuples. So take your pick, _LW, AbstractKeyedTuple, tuple would all be correct representations of what your function is returning.

SuperShoot
  • 9,880
  • 2
  • 38
  • 55
  • 12
    This is why we need [protocols](https://www.python.org/dev/peps/pep-0544/). – Ilja Everilä Mar 27 '19 at 06:50
  • `List[Tuple[int, str]]` may cause problems down the line, if one is then going to use the `_asdict()` method on each of the result's items: `Cannot access member "_asdict" for type "Tuple[str, str]"` – pmsoltani Sep 01 '20 at 13:22
  • This is a very good answer, however I want to add as of now _LW and AbstractKeyerTuple are no longer part of `sqlalchemy.util._collections`, might be changed or moved elsewhere – Matthias Nov 21 '22 at 14:41
  • 1
    @Matthias much has been done in sqlalchemy to make it more friendly for typing over the last year or two, so I suppose it’s not surprising to see some changes to the internals. There are going to be a lot of questions that need an update when 2.0 is released, I’ll try to find the time to update this answer. – SuperShoot Nov 21 '22 at 21:07
-1

Just print or log the type(entries) to see which type is used. No need to read through the module code.

Without checking it, the return might be the same as typical records you would get with cursor.fetchall(). Then, the type is just tuple - the Python built-in tuple. You do not need even need to import tuple to use tuple in the typing module.

Writing this without having tested it, yet, the main trick is anyway something else: use type(my_return_var) to see the type for the type hint.

Mind that a class with the module path must at first be imported.

Another example of how to use the "trick": type hint for a cursor object, taken from What is the right type hint for the returned sqlalchemy objects "cursor" and "cursor.fetchall()" records?. When the output of the type(...) is <class 'MySQLdb.cursors.Cursor'>, then you need

  • from MySQLdb.cursors import Cursor with Cursor as the type hint or
  • from MySQLdb import cursors with cursors.Cursor as the type hint or
  • import MySQLdb with MySQLdb.cursors.Cursor as the type hint.

This "trick" of getting the right type for typing is also at Type hints for SQLAlchemy engine and session objects.

questionto42
  • 7,175
  • 4
  • 57
  • 90
  • The whole point of this Q&A was that the output of `type(...)` resulted in `sqlalchemy.util._collections.result`, which was a dynamically constructed named tuple and something _not_ able to be imported like you suggest with this "trick". – SuperShoot Aug 27 '23 at 23:33
  • I guessed at the beginning that the type is likely a tuple, which it was, as you write it, and that this "tuple" type does not need to be imported. The "trick", on the other hand, was only about how to find out which type you have, not about what you need to import. – questionto42 Aug 28 '23 at 06:32