Joining with set-returning function (SRF) and access columns in SQLAlchemy

Question

Suppose I have an activity table and a subscription table. Each activity has an array of generic references to some other object, and each subscription has a single generic reference to some other object in the same set.

CREATE TABLE activity (
    id serial primary key,
    ob_refs UUID[] not null
);

CREATE TABLE subscription (
    id UUID primary key,
    ob_ref UUID,
    subscribed boolean not null
);

I want to join with the set-returning function unnest so I can find the "deepest" matching subscription, something like this:

SELECT id
FROM (
    SELECT DISTINCT ON (activity.id)
        activity.id,
        x.ob_ref, x.ob_depth,
        subscription.subscribed IS NULL OR subscription.subscribed = TRUE
            AS subscribed,
    FROM activity

    LEFT JOIN subscription
        ON activity.ob_refs @> array[subscription.ob_ref]

    LEFT JOIN unnest(activity.ob_refs)
        WITH ORDINALITY AS x(ob_ref, ob_depth)
        ON subscription.ob_ref = x.ob_ref

    ORDER BY x.ob_depth DESC
) sub
WHERE subscribed = TRUE;

But I can't figure out how to do that second join and get access to the columns. I've tried creating a FromClause like this:

act_ref_t = (sa.select(
        [sa.column('unnest', UUID).label('ob_ref'),
         sa.column('ordinality', sa.Integer).label('ob_depth')],
        from_obj=sa.func.unnest(Activity.ob_refs))
    .suffix_with('WITH ORDINALITY')
    .alias('act_ref_t'))

...

query = (query
    .outerjoin(
        act_ref_t,
        Subscription.ob_ref == act_ref_t.c.ob_ref))
    .order_by(activity.id, act_ref_t.ob_depth)

But that results in this SQL with another subquery:

LEFT OUTER JOIN (
        SELECT unnest AS ob_ref, ordinality AS ref_i
        FROM unnest(activity.ob_refs) WITH ORDINALITY
    ) AS act_ref_t
    ON subscription.ob_refs @> ARRAY[act_ref_t.ob_ref]

... which fails because of the missing and unsupported LATERAL keyword:

There is an entry for table "activity", but it cannot be referenced from this part of the query.

So, how can I create a JOIN clause for this SRF without using a subquery? Or is there something else I'm missing?

Edit 1 Using sa.text with TextClause.columns instead of sa.select gets me a lot closer:

act_ref_t = (sa.sql.text(
        "unnest(activity.ob_refs) WITH ORDINALITY")
    .columns(sa.column('unnest', UUID),
             sa.column('ordinality', sa.Integer))
    .alias('act_ref'))

But the resulting SQL fails because it wraps the clause in parentheses:

LEFT OUTER JOIN (unnest(activity.ob_refs) WITH ORDINALITY)
    AS act_ref ON subscription.ob_ref = act_ref.unnest

The error is syntax error at or near ")". Can I get TextAsFrom to not be wrapped in parentheses?

score 3 · Accepted Answer · edited May 23 '17 at 12:23

It turns out this is not directly supported by SA, but the correct behaviour can be achieved with a ColumnClause and a FunctionElement. First import this recipe as described by zzzeek in this SA issue. Then create a special unnest function that includes the WITH ORDINALITY modifier:

class unnest_func(ColumnFunction):
    name = 'unnest'
    column_names = ['unnest', 'ordinality']

@compiles(unnest_func)
def _compile_unnest_func(element, compiler, **kw):
    return compiler.visit_function(element, **kw) + " WITH ORDINALITY"

You can then use it in joins, ordering, etc. like this:

act_ref = unnest_func(Activity.ob_refs)
query = (query
    .add_columns(act_ref.c.unnest, act_ref.c.ordinality)
    .outerjoin(act_ref, sa.true())
    .outerjoin(Subscription, Subscription.ob_ref == act_ref.c.unnest)
    .order_by(act_ref.c.ordinality.desc()))

Joining with set-returning function (SRF) and access columns in SQLAlchemy

1 Answers1

Linked