Copy tables from one database to another in SQL Server, using Python

Question

Does anybody know of a good Python code that can copy large number of tables (around 100 tables) from one database to another in SQL Server?

I ask if there is a way to do it in Python, because due to restrictions at my place of employment, I cannot copy tables across databases inside SQL Server alone.

Here is a simple Python code that copies one table from one database to another. I am wondering if there is a better way to write it if I want to copy 100 tables.

print('Initializing...')

import pandas as pd
import sqlalchemy
import pyodbc

db1 = sqlalchemy.create_engine("mssql+pyodbc://user:password@db_one")
db2 = sqlalchemy.create_engine("mssql+pyodbc://user:password@db_two")

print('Writing...')

query = '''SELECT * FROM [dbo].[test_table]'''
df = pd.read_sql(query, db1)
df.to_sql('test_table', db2, schema='dbo', index=False, if_exists='replace')

print('(1) [test_table] copied.')

are you sure __all__ tables from the source DB will fit into RAM on the machine where you are going to run your Python script? — MaxU - stand with Ukraine, Oct 25 '17 at 21:35
store table names in a simple text file. Read the table name one at a time and then use the above script to copy one by one. You can print some info in between to log the progress. Even better, there can be a central machine which holds the list. You can have other worker machines which will read each table name individually and copy it that way you will distribute the work. But destination DB should be able to handle the incoming load as well. — Albatross, Oct 25 '17 at 21:35
@Confused can you show me a sample code or tell me how python would read in a text file? thank you so much!!! im seriously new to python — pynewbee, Oct 25 '17 at 21:38
Please refer (https://stackoverflow.com/questions/3277503/how-do-i-read-a-file-line-by-line-into-a-list). You can ignore the list part. — Albatross, Oct 26 '17 at 18:38

score 11 · Answer 1 · answered Oct 25 '17 at 21:49

11

SQLAlchemy is actually a good tool to use to create identical tables in the second db:

table = Table('test_table', metadata, autoload=True, autoload_with=db1)
table.create(engine=db2)

This method will also produce correct keys, indexes, foreign keys. Once the needed tables are created, you can move the data by either select/insert if the tables are relatively small or use bcp utility to dump table to disk and then load it into the second database (much faster but more work to get it to work correctly)

If using select/insert then it is better to insert in batches of 500 records or so.

answered Oct 25 '17 at 21:49

Muposat

1,476
1
11
24

2

`"This method will also produce correct keys, indexes, foreign keys."` +1 – MaxU - stand with Ukraine Oct 25 '17 at 21:50
I ditn't get the part of select/insert. How can I do this succintly with SQLAlchemy? – Mateus Felipe Jan 25 '18 at 11:43
@Mateus Felipe: something like this: `for rec in select(table1): table2.insert(rec)`. This is best if discussed as a separate question. Make sure to post a code sample if you do. – Muposat Jan 26 '18 at 00:00
@Muposat Thanks for your answer! I actually searched a little more and found this: https://stackoverflow.com/questions/21770829/sqlalchemy-copy-schema-and-data-of-subquery-to-another-database – Mateus Felipe Jan 26 '18 at 11:26

score 2 · Answer 2 · answered Oct 25 '17 at 21:44

You can do something like this:

tabs = pd.read_sql("SELECT table_name FROM INFORMATION_SCHEMA.TABLES", db1)

for tab in tabs['table_name']:
    pd.read_sql("select * from {}".format(tab), db1).to_sql(tab, db2, index=False)

But it might be be awfully slow. Use SQL Server tools to do this job.

Consider using sp_addlinkedserver procedure to link one SQL Server from another. After that you can execute:

SELECT * INTO server_name...table_name FROM table_name

for all tables from the db1 database.

PS this might be done in Python + SQLAlchemy as well...

Copy tables from one database to another in SQL Server, using Python

2 Answers2