need help optimizing wordpress meta_query

Question

I would some help in optimizing this wordpress query, it currently takes 100% cpu usage and have never got the chance for it to finish:

SELECT wp_posts.* FROM wp_posts INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)
INNER JOIN wp_postmeta AS mt1 ON (wp_posts.ID = mt1.post_id)
INNER JOIN wp_postmeta AS mt2 ON (wp_posts.ID = mt2.post_id)
INNER JOIN wp_postmeta AS mt3 ON (wp_posts.ID = mt3.post_id)
INNER JOIN wp_postmeta AS mt4 ON (wp_posts.ID = mt4.post_id)
INNER JOIN wp_postmeta AS mt5 ON (wp_posts.ID = mt5.post_id)
INNER JOIN wp_postmeta AS mt6 ON (wp_posts.ID = mt6.post_id)
INNER JOIN wp_postmeta AS mt7 ON (wp_posts.ID = mt7.post_id)
INNER JOIN wp_postmeta AS mt8 ON (wp_posts.ID = mt8.post_id)
INNER JOIN wp_postmeta AS mt9 ON (wp_posts.ID = mt9.post_id)
INNER JOIN wp_postmeta AS mt10 ON (wp_posts.ID = mt10.post_id) WHERE 1=1 AND wp_posts.post_type = 'produkter' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'private') AND (wp_postmeta.meta_key = 'csv_product_month_sub'
OR (mt1.meta_key = 'csv_product_type' AND CAST(mt1.meta_value AS CHAR) = 'Mobilt Bredbaand')
OR (mt2.meta_key = 'csv_product_consumption' AND CAST(mt2.meta_value AS SIGNED) BETWEEN '0' AND '2')
OR (mt3.meta_key = 'csv_product_consumption' AND CAST(mt3.meta_value AS SIGNED) BETWEEN '3' AND '9')
OR (mt4.meta_key = 'csv_product_consumption' AND CAST(mt4.meta_value AS SIGNED) BETWEEN '10' AND '19')
OR (mt5.meta_key = 'csv_product_download' AND CAST(mt5.meta_value AS SIGNED) BETWEEN '2' AND '9')
OR (mt6.meta_key = 'csv_product_download' AND CAST(mt6.meta_value AS SIGNED) BETWEEN '10' AND '19')
OR (mt7.meta_key = 'csv_product_download' AND CAST(mt7.meta_value AS SIGNED) BETWEEN '20' AND '29')
OR (mt8.meta_key = 'csv_product_month_sub' AND CAST(mt8.meta_value AS SIGNED) BETWEEN '0' AND '49')
OR (mt9.meta_key = 'csv_product_month_sub' AND CAST(mt9.meta_value AS SIGNED) BETWEEN '50' AND '99')
OR (mt10.meta_key = 'csv_product_month_sub' AND CAST(mt10.meta_value AS SIGNED) BETWEEN '100' AND '149') ) GROUP BY wp_posts.ID ORDER BY wp_postmeta.meta_value+0 ASC

First of all why you use INNER JOINs? And what is that ORDER meta_value+0 ? Why you make 3 ORs for the same meta_key check ? — Svetoslav, Oct 24 '12 at 04:27
this query is generated by WP_Query. There are 3 ORS per meta_key, because it checks for different ranges/values in each one. — , Oct 24 '12 at 04:31
I think that adding +0 to the ORDER BY, it "forces" mysql to use the value as a number, instead of text. — , Oct 24 '12 at 04:32

score 0 · Answer 1 · answered Oct 24 '12 at 06:04

So far I have:

SELECT wp_posts.* FROM wp_posts INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)
WHERE ID IN (
            SELECT post_id from wp_postmeta 
            WHERE 
            (
                (meta_key = 'csv_product_consumption' AND CAST(meta_value AS SIGNED) BETWEEN 0 AND 2) 
                OR (meta_key = 'csv_product_consumption' AND CAST(meta_value AS SIGNED) BETWEEN 3 AND 9) 
                OR (meta_key = 'csv_product_consumption' AND CAST(meta_value AS SIGNED) BETWEEN 10 AND 19) 
                OR (meta_key = 'csv_product_download' AND CAST(meta_value AS SIGNED) BETWEEN 2 AND 9) 
                OR (meta_key = 'csv_product_download' AND CAST(meta_value AS SIGNED) BETWEEN 10 AND 19) 
                OR (meta_key = 'csv_product_download' AND CAST(meta_value AS SIGNED) BETWEEN 20 AND 29) 
                OR (meta_key = 'csv_product_month_sub' AND CAST(meta_value AS SIGNED) BETWEEN 0 AND 49) 
                OR (meta_key = 'csv_product_month_sub' AND CAST(meta_value AS SIGNED) BETWEEN 50 AND 99) 
                OR (meta_key = 'csv_product_month_sub' AND CAST(meta_value AS SIGNED) BETWEEN 100 AND 149) 
            )
            GROUP BY post_id
        )
AND wp_posts.post_type = 'produkter' 
AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'private') 
AND ( (wp_postmeta.meta_key = 'csv_product_type') AND (wp_postmeta.meta_value = 'Mobilt Bredbaand'))
ORDER BY wp_postmeta.meta_value+0 ASC

It no longer uses too much cpu and the query takes 0.0331 sec.

Any other ideas would be appreciated.

score 0 · Answer 2 · answered Mar 13 '13 at 22:55

I came across this problem and realized the issue was with all the INNER JOINS generated by WordPress. I got the raw query from WordPress:

SELECT   wp_posts.* FROM wp_posts  
INNER JOIN wp_term_relationships ON (wp_posts.ID = wp_term_relationships.object_id) 
INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)
INNER JOIN wp_postmeta AS mt1 ON (wp_posts.ID = mt1.post_id)
INNER JOIN wp_postmeta AS mt2 ON (wp_posts.ID = mt2.post_id)
INNER JOIN wp_postmeta AS mt3 ON (wp_posts.ID = mt3.post_id)
INNER JOIN wp_postmeta AS mt4 ON (wp_posts.ID = mt4.post_id)
INNER JOIN wp_postmeta AS mt5 ON (wp_posts.ID = mt5.post_id) WHERE 1=1  AND ( wp_term_relationships.term_taxonomy_id IN (7,11,12,13,14,15) AND wp_posts.ID NOT IN (
                    SELECT object_id
                    FROM wp_term_relationships
                    WHERE term_taxonomy_id IN (10)
                ) ) AND wp_posts.post_type IN ('post', 'page', 'attachment', 'product', 'store_page') AND (wp_posts.post_status = 'publish') AND ( (wp_postmeta.meta_key = 'product_type' AND CAST(wp_postmeta.meta_value AS CHAR) = 'type_pre_ground')
OR  (mt1.meta_key = 'product_type2' AND CAST(mt1.meta_value AS CHAR) = 'type_pre_ground')
OR  (mt2.meta_key = 'product_type3' AND CAST(mt2.meta_value AS CHAR) = 'type_pre_ground')
OR  (mt3.meta_key = 'product_type4' AND CAST(mt3.meta_value AS CHAR) = 'type_pre_ground')
OR  (mt4.meta_key = 'product_type5' AND CAST(mt4.meta_value AS CHAR) = 'type_pre_ground')
OR  (mt5.meta_key = 'product_type6' AND CAST(mt5.meta_value AS CHAR) = 'type_pre_ground') ) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC

This never completed and tied up the CPU at a very high load. I removed the last two INNER JOINs (and corresponding WHERE clauses) and got results in 2 seconds:

SELECT   wp_posts.* FROM wp_posts  
INNER JOIN wp_term_relationships ON (wp_posts.ID = wp_term_relationships.object_id) 
INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)
INNER JOIN wp_postmeta AS mt1 ON (wp_posts.ID = mt1.post_id)
INNER JOIN wp_postmeta AS mt2 ON (wp_posts.ID = mt2.post_id)
INNER JOIN wp_postmeta AS mt3 ON (wp_posts.ID = mt3.post_id) WHERE 1=1  AND ( wp_term_relationships.term_taxonomy_id IN (7,11,12,13,14,15) AND wp_posts.ID NOT IN (
                    SELECT object_id
                    FROM wp_term_relationships
                    WHERE term_taxonomy_id IN (10)
                ) ) AND wp_posts.post_type IN ('post', 'page', 'attachment', 'product', 'store_page') AND (wp_posts.post_status = 'publish') AND ( (wp_postmeta.meta_key = 'product_type' AND CAST(wp_postmeta.meta_value AS CHAR) = 'type_pre_ground')
OR  (mt1.meta_key = 'product_type2' AND CAST(mt1.meta_value AS CHAR) = 'type_pre_ground')
OR  (mt2.meta_key = 'product_type3' AND CAST(mt2.meta_value AS CHAR) = 'type_pre_ground')
OR  (mt3.meta_key = 'product_type4' AND CAST(mt3.meta_value AS CHAR) = 'type_pre_ground') ) GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC

So I realized that removing the INNER JOINS was the key to speeding up the query. I re-wrote the query with only one wp_postmeta INNER JOIN and got results in fractions of a second:

SELECT   wp_posts.* FROM wp_posts  
INNER JOIN wp_term_relationships ON (wp_posts.ID = wp_term_relationships.object_id) 
INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id) WHERE 1=1  AND ( wp_term_relationships.term_taxonomy_id IN (7,11,12,13,14,15) AND wp_posts.ID NOT IN (
                    SELECT object_id
                    FROM wp_term_relationships
                    WHERE term_taxonomy_id IN (10)
                ) ) 
AND wp_posts.post_type IN ('post', 'page', 'attachment', 'product', 'store_page') 
AND (wp_posts.post_status = 'publish') 
AND ( (wp_postmeta.meta_key = 'product_type' AND CAST(wp_postmeta.meta_value AS CHAR) = 'type_pre_ground')
OR  (wp_postmeta.meta_key = 'product_type2' AND CAST(wp_postmeta.meta_value AS CHAR) = 'type_pre_ground')
OR  (wp_postmeta.meta_key = 'product_type3' AND CAST(wp_postmeta.meta_value AS CHAR) = 'type_pre_ground')
OR  (wp_postmeta.meta_key = 'product_type4' AND CAST(wp_postmeta.meta_value AS CHAR) = 'type_pre_ground')
OR  (wp_postmeta.meta_key = 'product_type5' AND CAST(wp_postmeta.meta_value AS CHAR) = 'type_pre_ground')
OR  (wp_postmeta.meta_key = 'product_type6' AND CAST(wp_postmeta.meta_value AS CHAR) = 'type_pre_ground') ) 
GROUP BY wp_posts.ID ORDER BY wp_posts.post_date DESC

With this solution I created a WordPress filter to convert the postmeta query to the faster format:

function custom_get_meta_sql( $meta_sql )
{
    global $wpdb;

    $posts_table = $wpdb->prefix . 'posts';
    $postmeta_table = $wpdb->prefix . 'postmeta';

    //use single INNER JOIN
    $meta_sql['join'] = " INNER JOIN {$postmeta_table} AS pmta ON ({$posts_table}.ID = pmta.post_id) ";

    //replace the mtNN aliases with wp_postmeta
    $where_clause = $meta_sql['where'];
    $where_clause = str_replace("{$postmeta_table}.", 'pmta.', $where_clause);
    $where_clause = preg_replace('/mt\d+\.meta_/i', 'pmta.meta_', $where_clause);

    $meta_sql['where'] = $where_clause;
    return $meta_sql;
}

add_filter( 'get_meta_sql', 'custom_get_meta_sql' );

This filter should be located in the functions.php file of your WordPress Theme.

thanks for sharing this. Maybe I'm missing something, but I can't see how this could work. Doesn't WordPress create multiple INNER JOINS to allow the SQL to select different meta_keys and meta_values on the same query? I am no SQL expert, but looking at the results it seems like if the WHERE clauses are joined by AND, this type of query will always return 0 results. This is because in the first condition we check if `pmta.meta_key = "x"` and in the second we check if `ptma.meta_key = "y"`. Both can't be true right? I would really love to be wrong about this though. What did I miss? — Dominic P, Mar 14 '13 at 19:49
The query would work since the AND is performed on the results of the inner ANDs: (KEY_CONDITION1 AND VALUE_CONDITION1) AND (KEY_CONDITION2 AND VALUE_CONDITION2). KEY_CONDITION1 doesn't have to agree with KEY_CONDITION2 since it is being tested against it's corresponding VALUE_CONDITION1. In your example, KEY_CONDITION1 is 'pmta.meta_key = "x"'. You need to also have VALUE_CONDITION1 in order to replicate what the query is testing. — KalenGi, Mar 15 '13 at 13:24
I can't understand why I'm getting 0 results with this then. I opened a new question with the SQL that's getting generated. Want to take a look? http://stackoverflow.com/q/15439474/931860 — Dominic P, Mar 15 '13 at 18:22
From the explanation given by @RichardSitze on your other question http://stackoverflow.com/a/15439659/212076, I agree that it won't work for AND. It worked in my case since what I needed was OR. This means the filter code that I've used would have to be structured in a way to avoid modifying queries that require AND. — KalenGi, Mar 16 '13 at 07:09

score 0 · Answer 3 · answered Nov 25 '13 at 20:26

This is an old post now, but you might consider trying Horizontal Meta. Horizontal Meta monitors specific keys in the post & user meta tables and extracts them out into a relational/horizontal format. You can still use the WordPress query engine to run queries, but Horizontal Meta will rewrite the queries to speed it up. Available here: http://wordpress.org/plugins/horizontal-meta/

need help optimizing wordpress meta_query

3 Answers3

Linked