Best way to insert multiple rows with asyncpg

Question

I want to insert multiple rows and get IDs back with asyncpg, i found two ways: 1: generate sql like this

INSERT INTO films (code, title, did, date_prod, kind) VALUES
    ('B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'),
    ('HG120', 'The Dinner Game', 140, DEFAULT, 'Comedy')
RETURNING id;

2: use prepared statement in for loop

values =(('B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'),
        ('HG120', 'The Dinner Game', 140, DEFAULT, 'Comedy'))
stmnt = connection.prepare("INSERT INTO films (code, title, did, date_prod, kind) VALUES $1, $2, $3, $4, $5  RETURNING id")
for val in values:
    stmnt.fetchval(*val)

which way i must prefer in case 100x times with 700 000 rows, or there is some way to combine this approaches? i totally green, so throw some tomattoes in me

You could try [`COPY FROM`](https://www.postgresql.org/docs/9.1/static/sql-copy.html). In my experience it's a lot faster than individual INSERT statements. — amphetamachine, May 02 '17 at 13:52
Possible duplicate of [psycopg2: insert multiple rows with one query](http://stackoverflow.com/questions/8134602/psycopg2-insert-multiple-rows-with-one-query) — Udi, May 02 '17 at 20:34

score 19 · Answer 1 · edited Nov 19 '19 at 00:38

19

asyncpg provides the executemany method to insert many rows.

statement = """INSERT INTO films (code,
                           title, 
                           did, 
                           date_prod, 
                           kind) VALUES($1, $2, $3, $4, $5);"""
await connection.executemany(statement, values)

If you need to use RETURNING as you later mentioned to return the inserted ids, this answer is the way to go.

edited Nov 19 '19 at 00:38

akkez

13
3

answered May 04 '17 at 10:15

styvane

59,869
19
150
156

thnx for right tag. i edit question because of ur answer - add "RETURNINNG" clause - so "executemany" now do not fit requirements, sorry for misslead. – 0sv4 May 05 '17 at 05:43
what if i have an array of objects, will this work? – PirateApp Aug 03 '22 at 11:47

Elvis Pranskevichus · Accepted Answer · 2017-05-12T13:59:33.560

If you need to use the RETURNING clause to obtain the ids back, then the following is the most efficient way of inserting multiple values:

res = await conn.fetch('''
    INSERT INTO films (code, title, did, date_prod, kind)
    (SELECT
        r.code, r.title, r.did, r.date_prod, r.kind
     FROM
        unnest($1::films[]) as r
    )
    RETURNING id
''', [
    (None, 'B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'),
    (None, 'HG120', 'The Dinner Game', 140, None, 'Comedy')
])

Note that the records you pass as input must correspond to the shape of the table: PostgreSQL does not support arbitrary records as input, so you must use a known record type. Simply pass the columns you are not inserting as None and don't include them in the SELECT return list. This method also doesn't allow you to rely on DEFAULT, you must specify each inserted value explicitly.

Just a minor note that "input must correspond to the shape of the table" is quite dangerous. Shape of the table, order of columns (verify with \d in psql) can vary from one PG server to another and this makes your code heavily dependent on how the tables were created on each db instance. — Shirish Kamath, Jun 07 '18 at 11:01

score 17 · Answer 3 · answered Apr 18 '19 at 19:00

17

Another way to insert many rows at once (assuming you don't need the inserted IDs) is to use the copy_records_to_table method.

data = [
    ("row", 1, "some data"),
    ("row", 2, "more data"),
]
await conn.copy_records_to_table('mytable', records=data)

answered Apr 18 '19 at 19:00

Thane Brimhall

9,256
7
36
50

1

Turns out this was the most convenient manner for me: I had batches of records that were grouped by their table name, and with their columns already in the correct order. – Matthew Schinckel Dec 10 '19 at 11:38
3

...although, I need "ON CONFLICT ... UPDATE ...", so I'm not sure I'll be able to use this mechanism. – Matthew Schinckel Dec 10 '19 at 11:43
1

instead of a plain tuple, would an array of objects do, how would you convert the object into a tuple if it doesnt – PirateApp Aug 03 '22 at 11:48
@PirateApp I don't see why an array wouldn't work, but if a tuple is strictly required, you can simply call `tuple(my_list)`. – Thane Brimhall Aug 03 '22 at 23:40

Best way to insert multiple rows with asyncpg

3 Answers3