503

I've got a very large MySQL table with about 150,000 rows of data. Currently, when I try and run

SELECT * FROM table WHERE id = '1';

the code runs fine as the ID field is the primary index. However, for a recent development in the project, I have to search the database by another field. For example:

SELECT * FROM table WHERE product_id = '1';

This field was not previously indexed; however, I've added one, so mysql now indexes the field, but when I try to run the above query, it runs very slowly. An EXPLAIN query reveals that there is no index for the product_id field when I've already added one, and as a result the query takes any where from 20 minutes to 30 minutes to return a single row.

My full EXPLAIN results are:

| id | select_type | table | type | possible_keys| key  | key_len | ref  | rows  | Extra       |
+----+-------------+-------+------+--------------+------+---------+------+-------+------------------+
|  1 | SIMPLE      | table | ALL  | NULL         | NULL | NULL    | NULL |157211 | Using where |
+----+-------------+-------+------+--------------+------+---------+------+-------+------------------+

It might be helpful to note that I've just taken a look, and ID field is stored as INT whereas the PRODUCT_ID field is stored as VARCHAR. Could this be the source of the problem?

Espoir Murhabazi
  • 5,973
  • 5
  • 42
  • 73
Michael
  • 5,505
  • 4
  • 20
  • 12
  • 2
    Can you post the full `EXPLAIN` results? Are you certain it's that there's **no** index? Or is the index there, but MySQL's choosing not to use it? – VoteyDisciple Jun 09 '10 at 01:44
  • 213
    A large table would have 150,000,000 records. A very large table has 15,000,000,000 records. A table of average size has 150,000. For future reference. – usumoio Oct 10 '12 at 17:56
  • 9
    Be aware that 'OR' can make MySql not use indexes. I had a query with 3 OR's. Each mached an index, and ran in 15ms, All together took between 25sec and timeout. So I made 3 queries and UNION'ed them together, it also took 15ms on 500.000 rows. – Leif Neland Mar 15 '17 at 13:34
  • Consider the data type you are storing. Performance may change based on your data type you are comparing. As you said PRODUCT_ID is a VARCHAR data type, try changing it to a INT and index the column. – gilbertdim Mar 24 '20 at 01:38

8 Answers8

768
ALTER TABLE `table` ADD INDEX `product_id_index` (`product_id`)

Never compare integer to strings in MySQL. If id is int, remove the quotes.

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
zerkms
  • 249,484
  • 69
  • 436
  • 539
  • 56
    Use `SHOW INDEXES FROM YOURTABLE` http://dev.mysql.com/doc/refman/5.0/en/show-index.html to check if the indexes have been added – Timo Huovinen Jun 07 '13 at 12:28
  • 8
    Today I had the exact problem @Michael describes, and the solution was to "Never compare integer to strings in mysql." Thank you. – user12345 Sep 10 '14 at 21:09
  • @zerkms `Never compare integer to strings in mysql` Why not? Doesn't it automatically convert strings to numbers in this case? – x-yuri Aug 14 '15 at 14:19
  • @x-yuri I have personally seen cases when it casted the wrong operand which caused the full scan. Not to say that it makes no sense. – zerkms Aug 14 '15 at 22:03
  • Well, [type conversion rules](https://dev.mysql.com/doc/refman/5.6/en/type-conversion.html) says that shouldn't happen. But I wouldn't vouch for that. As for "makes no sense", sure, if you write the statement yourself. But makes sense when it's generated. Why bother adapting to the type of a variable passed, if one could just convert it to string and let `mysql` handle it... – x-yuri Aug 14 '15 at 23:16
  • @x-yuri well, I did observe that really long ago, not sure what version was that. Even if you generate it automatically, it makes sense to implement a query generator that respects types. – zerkms Aug 14 '15 at 23:29
  • @ced: main reason why an index is not used, is cardinality and MySQL thinking a full table scan would be quicker than using the index, or another index might be better suited. Make sure your cardinalities are updated regularly (`ANALYZE TABLE` if need be), possibly increase their pages to get a more reliable number (at the expense of longer calculating those). If all that fails, and you are _really_ sure and _intimately_ know your data, the `USE / FORCE INDEX` are at your disposal, but that means you will be forced to check whether that works better every X amount of time from then on. – Wrikken Aug 20 '15 at 19:13
  • @Wrikken I replied above you probably didn't see. Actually the reason it wasn't used (or it might have been used idk) is that all my date field had the same value. I did enter programatically 100k values and I didn't think about that. – Ced Aug 21 '15 at 01:25
  • @Ced: yeah, just a drive-by answer when I saw someone commented on a comment more than 5 years old :) But: debugging this in general: if you would have checked cardinality, you would have seen your index was indeed estimated on only a few different values, making it indeed unappealing to use as in index, which stands to reason given how you filled it. So, the answer was more how to debug this in general rather then this specific case ;) – Wrikken Aug 23 '15 at 14:27
  • @Wrikken Yeah I usually check if the user has some reputation point before posting on an old comment to see if they are susceptible to answer. Anyway I'll have to check the carnality thing as I have no idea what that is now. Thanks for the tip. – Ced Aug 23 '15 at 20:00
  • Never compare integer to strings in MySQL. If id is int, remove the quotes. If product_id is interger, should I changed the alter table to ```ALTER TABLE `table` ADD INDEX `product_id_index` (product_id)``` – want_to_be_calm Jun 04 '20 at 09:52
  • @want_to_be_calm back tick `\`` is not the same as a single quote `'`. The former is used as an identifier punctuator, and the latter - for strings. – zerkms Jun 04 '20 at 23:17
179
ALTER TABLE TABLE_NAME ADD INDEX (COLUMN_NAME);
khr055
  • 28,690
  • 16
  • 36
  • 48
pabloferraz
  • 2,006
  • 1
  • 13
  • 7
  • 124
    In MySQL, if you use `ALTER TABLE tbl ADD INDEX (col)` instead of `ALTER TABLE tbl ADD INDEX col (col)`, then using `ALTER TABLE tbl ADD INDEX (col)` more than once will keep adding indices named `col_2`,`col_3`,... each time. Whereas using `ALTER TABLE tbl ADD INDEX col (col)` 2nd time, will give `ERROR 1061 (42000): Duplicate key name 'col'`. – Abhishek Oza May 21 '14 at 10:49
98

You can use this syntax to add an index and control the kind of index (HASH or BTREE).

create index your_index_name on your_table_name(your_column_name) using HASH;

or

create index your_index_name on your_table_name(your_column_name) using BTREE;

You can learn about differences between BTREE and HASH indexes here: http://dev.mysql.com/doc/refman/5.5/en/index-btree-hash.html

SwissCodeMen
  • 4,222
  • 8
  • 24
  • 34
Hieu Vo
  • 3,105
  • 30
  • 31
  • 1
    Hash converted to btree when I see using show indexes. – RN Kushwaha Oct 15 '15 at 06:18
  • 2
    What will be the default from Hash and BTree, if I don't specify any? – Bhavuk Mathur Apr 29 '16 at 08:39
  • 3
    @RNKushwaha because InnoDB and MyIsam don't support HASH, AFAIK, only Memory and NDB storage engines support it – Hieu Vo May 04 '16 at 07:56
  • To avoid duplicate indexes on the same columns, do not use this or `ALTER ... ADD INDEX (col)`, better use the `ALTER ... ADD INDEX col (col)` statement, see the accepted answer and the comment under the one that uses only `ADD INDEX (col)`. At least, when using `create ...`, I could make more than one index on the same column. Therefore, I guess that this answer has the same problem. Please comment if this is wrong. – questionto42 Apr 20 '22 at 09:36
75

Indexes of two types can be added: when you define a primary key, MySQL will take it as index by default.

Explanation

Primary key as index

Consider you have a tbl_student table and you want student_id as primary key:

ALTER TABLE `tbl_student` ADD PRIMARY KEY (`student_id`)

Above statement adds a primary key, which means that indexed values must be unique and cannot be NULL.

Specify index name

ALTER TABLE `tbl_student` ADD INDEX student_index (`student_id`)

Above statement will create an ordinary index with student_index name.

Create unique index

ALTER TABLE `tbl_student` ADD UNIQUE student_unique_index (`student_id`)

Here, student_unique_index is the index name assigned to student_id and creates an index for which values must be unique (here null can be accepted).

Fulltext option

ALTER TABLE `tbl_student` ADD FULLTEXT student_fulltext_index (`student_id`)

Above statement will create the Fulltext index name with student_fulltext_index, for which you need MyISAM Mysql Engine.

How to remove indexes ?

DROP INDEX `student_index` ON `tbl_student`

How to check available indexes?

SHOW INDEX FROM `tbl_student`
HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
Jazzzzzz
  • 1,593
  • 15
  • 16
71

It's worth noting that multiple field indexes can drastically improve your query performance. So in the above example we assume ProductID is the only field to lookup but were the query to say ProductID = 1 AND Category = 7 then a multiple column index helps. This is achieved with the following:

ALTER TABLE `table` ADD INDEX `index_name` (`col1`,`col2`)

Additionally the index should match the order of the query fields. In my extended example the index should be (ProductID,Category) not the other way around.

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
Antony
  • 3,875
  • 30
  • 32
18

You say you have an index, the explain says otherwise. However, if you really do, this is how to continue:

If you have an index on the column, and MySQL decides not to use it, it may by because:

  1. There's another index in the query MySQL deems more appropriate to use, and it can use only one. The solution is usually an index spanning multiple columns if their normal method of retrieval is by value of more then one column.
  2. MySQL decides there are to many matching rows, and thinks a tablescan is probably faster. If that isn't the case, sometimes an ANALYZE TABLE helps.
  3. In more complex queries, it decides not to use it based on extremely intelligent thought-out voodoo in the query-plan that for some reason just not fits your current requirements.

In the case of (2) or (3), you could coax MySQL into using the index by index hint sytax, but if you do, be sure run some tests to determine whether it actually improves performance to use the index as you hint it.

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
Wrikken
  • 69,272
  • 8
  • 97
  • 136
2

A better option is to add the constraints directly during CREATE TABLE query (assuming you have the information about the tables)

CREATE TABLE products(
    productId INT AUTO_INCREMENT PRIMARY KEY,
    productName varchar(100) not null,
    categoryId INT NOT NULL,
    CONSTRAINT fk_category
    FOREIGN KEY (categoryId) 
    REFERENCES categories(categoryId)
        ON UPDATE CASCADE
        ON DELETE CASCADE
) ENGINE=INNODB;
Dharman
  • 30,962
  • 25
  • 85
  • 135
Ashwin Balani
  • 745
  • 7
  • 19
-3

use phpmyadmin, great tool for MySQL managing, include indexing

mostapha
  • 189
  • 1
  • 7
  • 1
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/30835385) – Nico Haase Jan 20 '22 at 19:54