Fetch table values using alembic and update to another table.

Question

I have oauth secret and oauth key in client table. Now I moving them to oauth credentials table which will be created during migration. Alembic produced following schema for upgrade.

from myapp.models import Client, ClientCredential
from alembic import op
import sqlalchemy as sa


def upgrade():
### commands auto generated by Alembic - please adjust! ###
    op.create_table('client_credential',
    sa.Column('id', sa.Integer(), nullable=False),
    sa.Column('created_at', sa.DateTime(), nullable=False),
    sa.Column('updated_at', sa.DateTime(), nullable=False),
    sa.Column('client_id', sa.Integer(), nullable=False),
    sa.Column('key', sa.String(length=22), nullable=False),
    sa.Column('secret', sa.String(length=44), nullable=False),
    sa.Column('is_active', sa.Boolean(), nullable=False),
    sa.ForeignKeyConstraint(['client_id'], ['client.id'], ),
    sa.PrimaryKeyConstraint('id'),
    sa.UniqueConstraint('key')
   )
   # Here I need to copy data from table A to newly created Table.
   # Now Client table will not have secret and key attributes
   clients = [{'secret': client.secret, 'key': client.key, 'is_active':True, 'client_id': client.id, 'created_at': sa.func.now(), 'updated_at': sa.func.now()} for client in Client.query.all()]
   op.bulk_insert(ClientCredential, clients)
   #Also replaced above two lines with 
   #connection = op.get_bind()
   #print connection.execute(Client, Client.query.all())
   op.drop_column(u'client', u'secret')
   op.drop_column(u'client', u'key')

Once script enters clients or connection.execute alembic script hangs. After enabling sqlalchemy logs, Base.Engine is empty. Also tried op.execute, no luck.

Log

INFO  [alembic.migration] Context impl PostgresqlImpl.
INFO  [alembic.migration] Will assume transactional DDL.
INFO  [alembic.migration] Running upgrade 25e7a9839cd4 -> 176fb70348b9, Added  ClientCredential
2013-09-10 23:59:08,144 INFO sqlalchemy.engine.base.Engine select version()
INFO  [sqlalchemy.engine.base.Engine] select version()
2013-09-10 23:59:08,145 INFO sqlalchemy.engine.base.Engine {}
INFO  [sqlalchemy.engine.base.Engine] {}
2013-09-10 23:59:08,146 INFO sqlalchemy.engine.base.Engine select current_schema()
INFO  [sqlalchemy.engine.base.Engine] select current_schema()
2013-09-10 23:59:08,146 INFO sqlalchemy.engine.base.Engine {}
INFO  [sqlalchemy.engine.base.Engine] {}
2013-09-10 23:59:08,148 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
INFO  [sqlalchemy.engine.base.Engine] BEGIN (implicit)
2013-09-10 23:59:08,150 INFO sqlalchemy.engine.base.Engine SELECT client.id AS client_id,    client.created_at AS client_created_at, client.updated_at AS client_updated_at, client.user_id   AS client_user_id, client.org_id AS client_org_id, client.title AS client_title,   client.description AS client_description, client.website AS client_website, client.redirect_uri AS client_redirect_uri, client.notification_uri AS   client_notification_uri, client.iframe_uri AS client_iframe_uri, client.resource_uri AS client_resource_uri, client.active AS client_active, client.allow_any_login AS client_allow_any_login, client.team_access AS client_team_access, client.trusted AS client_trusted
FROM client
INFO  [sqlalchemy.engine.base.Engine] SELECT client.id AS client_id, client.created_at AS    client_created_at, client.updated_at AS client_updated_at, client.user_id AS client_user_id, client.org_id AS client_org_id, client.title AS client_title, client.description AS client_description, client.website AS client_website, client.redirect_uri AS client_redirect_uri, client.notification_uri AS client_notification_uri, client.iframe_uri AS client_iframe_uri, client.resource_uri AS client_resource_uri, client.active AS client_active, client.allow_any_login AS client_allow_any_login, client.team_access AS client_team_access, client.trusted AS client_trusted
FROM client
2013-09-10 23:59:08,150 INFO sqlalchemy.engine.base.Engine {}
INFO  [sqlalchemy.engine.base.Engine] {}

How can I copy the values from client table to client_credential table using alembic migration?

See http://stackoverflow.com/questions/15725859/sqlalchemy-alembic-bulk-insert-fails-str-object-has-no-attribute-autoincre and http://stackoverflow.com/questions/15543613/alembic-and-getting-the-last-inserted-value, may help. — alecxe, Sep 10 '13 at 19:10

score 61 · Accepted Answer · answered Sep 11 '13 at 10:56

Finally I solved the problem. Created raw sql to fetch the values and used bulk_insert.

def upgrade():
    ### commands auto generated by Alembic - please adjust! ###
    op.create_table('client_credential',
    sa.Column('id', sa.Integer(), nullable=False),
    sa.Column('created_at', sa.DateTime(), nullable=False),
    sa.Column('updated_at', sa.DateTime(), nullable=False),
    sa.Column('client_id', sa.Integer(), nullable=False),
    sa.Column('key', sa.String(length=22), nullable=False),
    sa.Column('secret', sa.String(length=44), nullable=False),
    sa.Column('is_active', sa.Boolean(), nullable=False),
    sa.ForeignKeyConstraint(['client_id'], ['client.id'], ),
    sa.PrimaryKeyConstraint('id'),
    sa.UniqueConstraint('key')
    )
    #http://stackoverflow.com/questions/15725859/sqlalchemy-alembic-bulk-insert-fails-str-object-has-no-attribute-autoincre
    client_credential = sa.sql.table('client_credential',
        sa.Column('client_id', sa.Integer, nullable=False),
        sa.Column('is_active', sa.Boolean, nullable=False, default=True),
        sa.Column('key', sa.String(22), nullable=False, default=True),
        sa.Column('secret', sa.String(22), nullable=False, default=True),
        sa.Column('created_at', sa.DateTime, nullable=False, default=sa.func.now()),
        sa.Column('updated_at', sa.DateTime, nullable=False, default=sa.func.now()),
    )
    conn = op.get_bind()
    res = conn.execute("select secret, key, id from client")
    results = res.fetchall()
    clients = [{'secret': r[0], 'key': r[1], 'is_active':True, 'client_id': r[2], 'created_at': datetime.datetime.now(), 'updated_at': datetime.datetime.now()} for r in results]
    op.bulk_insert(client_credential, clients)
    op.drop_column(u'client', u'secret')
    op.drop_column(u'client', u'key')
    ### end Alembic commands ###

As of alembic 0.7 (released 2014), defining the table twice isn't needed, since op.create_table() returns the created table definition. — dequis, Aug 23 '16 at 23:01

score 3 · Answer 2 · edited Mar 28 '14 at 10:50

You can create a DBSession with sqlalchemy engine bind, then you can avoid use SQL query.

from myapp.models import Client, ClientCredential
from alembic import op, context
import sqlalchemy as sa


def upgrade():
    ### commands auto generated by Alembic - please adjust! ###
    url = context.config.get_main_option("sqlalchemy.url")
    engine = sa.create_engine(url)
    DBSession.configure(bind=engine)

    op.create_table(
        'client_credential',
        sa.Column('id', sa.Integer(), nullable=False),
        sa.Column('created_at', sa.DateTime(), nullable=False),
        sa.Column('updated_at', sa.DateTime(), nullable=False),
        sa.Column('client_id', sa.Integer(), nullable=False),
        sa.Column('key', sa.String(length=22), nullable=False),
        sa.Column('secret', sa.String(length=44), nullable=False),
        sa.Column('is_active', sa.Boolean(), nullable=False),
        sa.ForeignKeyConstraint(['client_id'], ['client.id'], ),
        sa.PrimaryKeyConstraint('id'),
        sa.UniqueConstraint('key'))
    # Here I need to copy data from table A to newly created Table.
    # Now Client table will not have secret and key attributes
    clients = [
        {'secret': client.secret,
         'key': client.key,
         'is_active': True,
         'client_id': client.id,
         'created_at': sa.func.now(),
         'updated_at': sa.func.now()}
        for client in Client.query.all()]
    op.bulk_insert(ClientCredential, clients)
    #Also replaced above two lines with
    #connection = op.get_bind()
    #print connection.execute(Client, Client.query.all())
    op.drop_column(u'client', u'secret')
    op.drop_column(u'client', u'key')

when importing from your codebase, you risk that future changes compromise your older migrations. For example, if you decide later to rename 'secret' to 'sharedsecret', this migration will fail. Imho, migrations should be self-contained. — keppla, Sep 04 '14 at 14:33
@keppla: I am currently struggling with the same problem as kracekumar. You say that migrations should be self-contained, but how then would you solve the problem of moving data to a different table. Is that a kind of migration you would just not advice doing? Or would you advise to do it with raw sql as then you don't depend on the source code structure? — Dolf Andringa, Jun 01 '15 at 09:32
@DolfAndringa by selfcontained, i meant that all a migration needs to function must be part of the migration. In your example, the second table definition should be part of the migration too. It should not just be imported from your model, because maybe in three migrations you delete that table and your migration fails with in import error — keppla, Jun 01 '15 at 12:12

score 0 · Answer 3 · answered Oct 27 '18 at 19:22

Maybe try adding a column_reflect listener? E.g.:

def listen_for_reflect(inspector, table, column_info):
    "correct an ENUM type"
    if column_info['name'] == 'my_enum':
        column_info['type'] = Enum('a', 'b', 'c')

with self.op.batch_alter_table(
    "bar",
    reflect_kwargs=dict(
        listeners=[
            ('column_reflect', listen_for_reflect)
        ]
    )
) as batch_op:
    batch_op.alter_column(
        'flag', new_column_name='bflag', existing_type=Boolean)

More info: https://alembic.zzzcomputing.com/en/latest/batch.html#controlling-table-reflection

Fetch table values using alembic and update to another table.

3 Answers3

Linked