46

I have table structure like this

enter image description here

when I insert row to the table I'm using this query:

INSERT INTO table_blah ( material_item, ... hidden ) VALUES ( data, ... data ) ON DUPLICATE KEY UPDATE id = id, material_item = data, ... hidden = data;

when I first insert data without triggering the ON DUPLICATE KEY the id increments fine:

enter image description here

but when the ON DUPLICATE KEY triggers and i INSERT A NEW ROW the id looks odd to me:

enter image description here

How can I keep the auto increment, increment properly even when it triggers ON DUPLICATE KEY?

Arslan Ali
  • 17,418
  • 8
  • 58
  • 76
newbie
  • 1,884
  • 7
  • 34
  • 55

10 Answers10

60

This behavior is documented (paragraph in parentheses):

If you specify ON DUPLICATE KEY UPDATE, and a row is inserted that would cause a duplicate value in a UNIQUE index or PRIMARY KEY, MySQL performs an UPDATE of the old row. For example, if column a is declared as UNIQUE and contains the value 1, the following two statements have similar effect:

    INSERT INTO table (a,b,c) VALUES (1,2,3)   ON DUPLICATE KEY UPDATE c=c+1;

    UPDATE table SET c=c+1 WHERE a=1;

(The effects are not identical for an InnoDB table where a is an auto-increment column. With an auto-increment column, an INSERT statement increases the auto-increment value but UPDATE does not.)

Here is a simple explanation. MySQL attempts to do the insert first. This is when the id gets auto incremented. Once incremented, it stays. Then the duplicate is detected and the update happens. But the value gets missed.

You should not depend on auto_increment having no gaps. If that is a requirement, the overhead on the updates and inserts is much larger. Essentially, you need to put a lock on the entire table, and renumber everything that needs to be renumbered, typically using a trigger. A better solution is to calculate incremental values on output.

danronmoon
  • 3,814
  • 5
  • 34
  • 56
Gordon Linoff
  • 1,242,037
  • 58
  • 646
  • 786
  • 3
    +1. Well said. Thanks for including the actual documentation. ;-) – ghoti May 07 '14 at 12:01
  • If auto_increment isn't a requirement do you think the gap is ok? – newbie May 07 '14 at 12:15
  • @newbie . . . Use `auto_increment` for what it is intended for, which is to provide an increasing sequence of values as rows are inserted. If you need something else, use an appropriate mechanism, such as calculating the values on output or using triggers. – Gordon Linoff May 07 '14 at 14:11
  • 41
    This is stupid issue with mysql. Ok, I don't relay on it to be without gaps, but today I got hit limit of 16000000+ row (MEDIUMINT) when I have only around 500000 records which got often updated. Problem is that I have another tables with goes beyond 250Gb of size linked to this one - and to update it to INT (and loose storage space when its totally unneeded) takes days. – lapkritinis Mar 27 '17 at 09:59
  • 22
    I don't understand the reasoning behind this design. While I don't mind the gaps, I see no reason for the autoincrementation to occur as the ID never existed, so it just creates a lot of garbage IDs for nothing. – jsmars Feb 26 '18 at 13:52
  • 5
    I'm updating multiple records regularly using `ON DUPLICATE KEY UPDATE `. I don't care for the gaps but I fear from hitting the max column value. will using `ALTER TABLE table_name AUTO_INCREMENT=1` (which resets the auto_increment to the last rows id + 1) regularly after each batch update be a bad practice? – David Avikasis Oct 03 '18 at 21:48
7

This question is a fairly old one, but I answer it maybe it helps someone, to solve the auto-incrementing problem use the following code before insert/on duplicate update part and execute them all together:

SET @NEW_AI = (SELECT MAX(`the_id`)+1 FROM `table_blah`);
SET @ALTER_SQL = CONCAT('ALTER TABLE `table_blah` AUTO_INCREMENT =', @NEW_AI);
PREPARE NEWSQL FROM @ALTER_SQL;
EXECUTE NEWSQL; 

together and in one statement it should be something like below:

SET @NEW_AI = (SELECT MAX(`the_id`)+1 FROM `table_blah`);
SET @ALTER_SQL = CONCAT('ALTER TABLE `table_blah` AUTO_INCREMENT =', @NEW_AI);
PREPARE NEWSQL FROM @ALTER_SQL;
EXECUTE NEWSQL; 
INSERT INTO `table_blah` (`the_col`) VALUES("the_value")
ON DUPLICATE KEY UPDATE `the_col` = "the_value";
  • 4
    All the answers like "don't worry about gaps" are garbage, this should be accepted answer. I was running out of numbers for ID, I used the code above, for INSERT with many values added at once and it worked beautifully. Thank you ;) – Dariusz Chowański Jun 23 '20 at 14:06
  • The `id` stays the same one, but the operation still returns the affected rows count as normal `bug` way. – Lebecca Nov 09 '20 at 06:12
  • I agree with @DariuszChowański. This should be the accepted answer. With these 4 lines you no longer have gaps. – user1274113 Sep 22 '22 at 19:41
  • I put another answer using php. But I see this is pure sql? Can you explain how it works? What is the AI part mean in your NEW_AI variable? I don't know what concat is, you are setting the autoincrement value? I dont understand Prepeare or Execute commands either. If you can of commented between each line explaining the step, this could help other people understand your answer. However this is clearly a better answer than my suggestion. – Rosski Nov 11 '22 at 13:05
  • This answer doesn't consider concurrency or batch inserts. – Paul Spiegel Apr 21 '23 at 14:31
  • @Rosski AI stands for auto-increment. It finds the biggest existing Id, increments it by one and considers it as the start point for the table's auto-increment. – ali asghar tofighian Jun 11 '23 at 08:19
3

Change database engine from InnoDB to MyIsam will resolve your issue.

Delickate
  • 1,102
  • 1
  • 11
  • 17
2

I had the same frustration of gaps in the auto increment but I found a way to avoid it.

In terms of previouslly discussed "overheads". When I first wrote my DB query code, it did so many separate queries that it took 5 hours. Once I put on "ON DUPLICATE KEY UPDATE" it got it down to about 50 seconds. Amazing! Anyway the way I solved it was by using 2 queries. Which doulbles the time it takes to 2 minutes, which is still fine.

First I did an sql query for writing all the data (updates and inserts), but I included "IGNORE" in the first query, so this just bypasses the updates and only inserts the new stuff. So assuming your auto_increment previously has no gaps then it will still have no gaps because its only new records. I believe it is updates that cause the gaps. So for inserts: "INSERT IGNORE INTO mytablename(stuff,stuff2) VALUES "

Next I did the "ON DUPLICATE KEY UPDATE" variation of that sql query. It will keep the ID's in tact because all the records being updated have ID's already. The only thing it breaks is the auto_increment value, which gets incremented when a new record is added (or updated). So the solution is to patch this auto_increment value back to what it was before, once you have applied the updates.

To patch the auto increment value use this sql in your php: "ALTER TABLE mytablename AUTO_INCREMENT = " . ($TableCount + 1);

This works because when you do the updates you are not increasing the amount of records. Therefore we can use the tablecount to know what the next ID should be. You set $TableCount to the table count, then we add 1 and that's the next auto increment number.

This is cheap and dirty but it seems to work. Could be bad using this while something else is writing to the db though.

Rosski
  • 62
  • 4
  • It is safer to use $TableMax instead of $TableCount same as @aliasghartofighian answer above (some rows might've been deleted for example). – fresher96 Oct 23 '22 at 19:27
  • well my example was not pure sql, I was using php. TableMax and TableCount is kind of the same thing just whatever variable name makes more sense to the coder. I came from VB language so for me a count was the amount of items. I don't actually understand his code because I am not as good as him. But maybe I will ask him to elaborate out of interest. – Rosski Nov 11 '22 at 13:01
1

I often deal with this by creating a temporary table, recording in the temporary table whether the record is new or not, doing an UPDATE only on the rows that are not new, and doing an INSERT with the new rows. Here's a complete example:

## THE SETUP

# This is the table we're trying to insert into
DROP TABLE IF EXISTS items;
CREATE TABLE items (
  id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
  name VARCHAR(100) UNIQUE,
  price INT
);

# Put a few rows into the table
INSERT INTO items (name, price) VALUES
  ("Bike", 200),
  ("Basketball", 10),
  ("Fishing rod", 25)
;

## THE INSERT/UPDATE

# Create a temporary table to help with the update
DROP TEMPORARY TABLE IF EXISTS itemUpdates;
CREATE TEMPORARY TABLE itemUpdates (
  name VARCHAR(100) UNIQUE,
  price INT,
  isNew BOOLEAN DEFAULT(true)
);

# Change the price of the Bike and Basketball and add a new Tent item
INSERT INTO itemUpdates (name, price) VALUES
  ("Bike", 150),
  ("Basketball", 8),
  ("Tent", 100)
;

# For items that already exist, set isNew false
UPDATE itemUpdates
JOIN items
ON items.name = itemUpdates.name
SET isNew = false;

# UPDATE the already-existing items
UPDATE items
JOIN itemUpdates
ON items.name = itemUpdates.name
SET items.price = itemUpdates.price
WHERE itemUpdates.isNew = false;

# INSERT the new items
INSERT INTO items (name, price)
SELECT name, price
FROM itemUpdates
WHERE itemUpdates.isNew = true;

# Check the results
SELECT * FROM items;
# Results:
# ID | Name        | Price
# 1  | Bike        | 150
# 2  | Basketball  | 8
# 3  | Fishing rod | 25
# 4  | Tent        | 100
        

The INSERT IGNORE INTO approach is simpler, but it ignores any error, which isn't what I want. And I agree that this is strange behavior on the part of MySQL but it's what we've got to work with.

Aurast
  • 3,189
  • 15
  • 24
0

I just thought I'd add, as i was trying to find an answer to my problem. I could not stop the duplicate warning and found it was because I had it set it to TINYINT which only allows 127 entries, changing to SMALL/MED/BIGINT allows for many more

Naxia
  • 9
  • 4
0

A simple solution can be setting the id using the max(id) of the table.

INSERT INTO my_table (id, value) VALUES (
 (SELECT MAX(id) FROM my_table x) + 1, 
 'some unique value'
) 
ON DUPLICATE KEY UPDATE email = VALUE(email)
  • warning, this is not legal in InnoDB, as you cannot use subquery SELECT from same table you're INSERTING to, and InnoDB is the default since MySQL 5.5.5 released in 2010. – hanshenrik May 23 '23 at 22:00
-1

I don't think this is a problem with MySQL 5.6. See this example.

yyFred
  • 775
  • 9
  • 13
-1

ON DUPLICATE KEY UPDATE id=LAST_INSERT_ID(id)

-3

INSERT INTO table_blah ( material_item, ... hidden ) VALUES ( data, ... data ) ON DUPLICATE KEY UPDATE material_item = data, ... hidden = data

Yes remove the ID=ID as it will automaticly add where PRIMARY KEY = PRIMARY KEY...

Seti
  • 2,169
  • 16
  • 26