1

I have a big problem with the execution of a MySql query that is very slow.. too slow... unusable!

To read 1000 products with their prices it takes more than 20 seconds!!!

$dati = mysqli_query($mysqli_connect, "
SELECT      *  
FROM        $tb_products
LEFT JOIN   $tb_categories ON $tb_products.product_category = $tb_categories.category_id_master
LEFT JOIN   $tb_subcategories ON $tb_products.product_subcategory = $tb_subcategories.subcategory_id_master
LEFT JOIN   $tb_logos ON $tb_products.product_logo = $tb_logos.logo_id_master

LEFT JOIN   $tb_prices ON ( 
            $tb_products.product_brand = $tb_prices.price_brand
            AND $tb_products.product_code = $tb_prices.price_code
            AND $tb_prices.price_validity = (
                SELECT  MAX($tb_prices.price_validity) 
                FROM    $tb_prices 
                WHERE   $tb_prices.price_validity<=DATE_ADD(CURDATE(), INTERVAL +0 DAY)
                        AND $tb_products.product_code = $tb_prices.price_code
            )
        )

WHERE       $tb_products.product_language='$activeLanguage' AND $tb_products.product_category!=0
GROUP BY    $tb_products.product_code
ORDER BY    $tb_products.product_brand, $tb_categories.category_rank, $tb_subcategories.subcategory_rank, $tb_products.product_subcategory, $tb_products.product_rank
");

EDIT:
I've changed, as suggested from Mr.Alvaro, the SELECT * with a more efficient SELECT [list of values] and the execution time dropped from 20 seconds to 14 seconds. Still too slow...
END EDIT

Each product can have different prices, so I use the (select max...) to take the most recent (but not future) price. Maybe is this function that slow down everything? Are there better solutions in your opinion?

Consider that the same query without the join with the prices it takes only 0.2 seconds. So I'm convinced that the problem is all in that part of the code.

$dati = mysqli_query($mysqli_connect, "
SELECT      *  
FROM        $tb_products
LEFT JOIN   $tb_categories ON $tb_products.product_category = $tb_categories.category_id_master
LEFT JOIN   $tb_subcategories ON $tb_products.product_subcategory = $tb_subcategories.subcategory_id_master
LEFT JOIN   $tb_logos ON $tb_products.product_logo = $tb_logos.logo_id_master

WHERE       $tb_products.product_language='$activeLanguage' AND $tb_products.product_category!=0
GROUP BY    $tb_products.product_code
ORDER BY    $tb_products.product_brand, $tb_categories.category_rank, $tb_subcategories.subcategory_rank, $tb_products.product_subcategory, $tb_products.product_rank
");

I also considered the fact that it could depend on the power of the server but I would tend to exclude it because the second query (without prices) is quite acceptable as speed.

The prices table is as following

+----------------+-------------+
| price_id       | int(3)      |
| price_brand    | varchar(5)  |
| price_code     | varchar(50) |
| price_value    | float(10,2) |
| price_validity | date        |
| price_language | varchar(2)  |
+----------------+-------------+
  • Which database + version? If I remember correctly, there was a significant performance boost between MySQL 5.5.21 and 5.6.4. – RAZERZ Dec 04 '17 at 11:08
  • I think the line `WHERE $tb_prices.price_validity<=DATE_ADD(CURDATE(), INTERVAL +0 DAY)` is slowing down because it needs to calculate this for every line. Or maybe you need to add some indexes to your table – Phate01 Dec 04 '17 at 11:14
  • Php: 5.6.30 - MySql: 5.6.38 – ActiveMedia Dec 04 '17 at 11:17
  • Just for testing purpouses using @Phate01 suggestion, run this query by itself and check the time. SELECT MAX($tb_prices.price_validity) FROM $tb_prices WHERE $tb_prices.price_validity<=DATE_ADD(CURDATE(), INTERVAL +0 DAY) AND $tb_products.product_code = $tb_prices.price_code; and check the time to perform this one. – Mr.Alvaro Dec 04 '17 at 12:29
  • If I use SELECT MAX() the query doesn't work.. but if I use only SELECT $tb_prices.price_validity..... it takes 0.006 seconds. Obviously this query read only 1 product – ActiveMedia Dec 04 '17 at 14:40
  • The thing is we need to figure out what part of the query is the one that is taking a long time to process. After your statement on the question is obvious is the last LEFT JOIN and that LEFT JOIN have a subquery, so we need to deduct going backwards. If you check the last LEFT JOIN, there is one condition that say "AND $tb_prices.price_validity = (SELECT..." so try to run that last part of the query by itself. "SELECT MAX($tb_prices.price..." and let us know how long it takes to show the result. – Mr.Alvaro Dec 04 '17 at 16:05
  • Are you using `LEFT` because the category and subcategory, etc, are optional? Remove `LEFT` otherwise. – Rick James Dec 04 '17 at 19:18
  • Please provide `EXPLAIN SELECT ... ` – Rick James Dec 04 '17 at 19:18
  • For money columns, use `DECIMAL`, _not_ `FLOAT`. – Rick James Dec 04 '17 at 19:20
  • What indexes do you have? For starters, what about `category_id_master`? – Rick James Dec 04 '17 at 20:39
  • @Mr.Alvaro.. I'll try again with the SELECT MAX... – ActiveMedia Dec 05 '17 at 08:26
  • @Rick James.. I'm not sure how to use EXPLAIN and how to dump the results... I've tried var_dump($results) and it return this: `object(mysqli_result)#2 (5) { ["current_field"]=> int(0) ["field_count"]=> int(72) ["lengths"]=> NULL ["num_rows"]=> int(1460) ["type"]=> int(0) }` – ActiveMedia Dec 05 '17 at 08:28
  • Nothing... I'm not able to run only the SELECT MAX(...) It give me no results! – ActiveMedia Dec 05 '17 at 08:54
  • If you ran the query with and without the last left join do you see any difference in the result? Because if you say you are getting no results from the SELECT MAX that mean that the whole LEFT JOIN is not doing anything because those AND are returning FALSE. – Mr.Alvaro Dec 05 '17 at 15:26
  • Please learn to use the commandline mysql tool. – Rick James Dec 05 '17 at 17:54
  • Hi guys, I solved! I managed to execute the SELECT MAX(..) separately and it takes 0.1 seconds to execute. So the problem was the JOIN (boy I still don't know the reason). Now I add an answer with full explanation. Thank you all! – ActiveMedia Dec 06 '17 at 11:44

2 Answers2

2

Maybe is because you are using SELECT *, this is known as bad practice. Check this question in stack overflow.

Is there a difference between Select * and Select [list each col]

There, Mitch Wheat wrote:

You should specify an explicit column list. SELECT * will bring back more columns than you need creating more IO and network traffic, but more importantly it might require extra lookups even though a non-clustered covering index exists (On SQL Server). Blockquote

Mr.Alvaro
  • 99
  • 10
  • Thanks for the suggestion! I've changed the query using a list of select and the execution time fell from 20 s to 14 s. However it is still too slow .. I have to find other solutions .. – ActiveMedia Dec 04 '17 at 11:03
  • Can you tell us the hardware configuration you are using for the DB server, specially the storage related. – Mr.Alvaro Dec 04 '17 at 12:21
2

SOLVED

The problem was in the last JOIN with the prices table. Following the suggestions I managed to execute the SELECT MAX (...) separately and it tooks 0.1 seconds to execute.

So I decide to run the main query without the prices and then, in the WHILE cicle to fetch the array, I run a second query to take the price for every single product! This work perfectly and my page has dropped down from 20 seconds to a few tenths of a second.

So, the code become something like this:

$dati = mysqli_query($mysqli_connect, "
SELECT      *  
FROM        $tb_products
LEFT JOIN   $tb_categories ON $tb_products.product_category =     $tb_categories.category_id_master
LEFT JOIN   $tb_subcategories ON $tb_products.product_subcategory =     $tb_subcategories.subcategory_id_master
LEFT JOIN   $tb_logos ON $tb_products.product_logo = $tb_logos.logo_id_master

WHERE       $tb_products.product_language='$activeLanguage' AND     $tb_products.product_category!=0
GROUP BY    $tb_products.product_code
ORDER BY    $tb_products.product_brand, $tb_categories.category_rank,     $tb_subcategories.subcategory_rank, $tb_products.product_subcategory,     $tb_products.product_rank
");

and then..

while ($array = mysqli_fetch_array($dati)) {

    $code = $array['product_code'];

    $dati_prices = mysqli_query($mysqli_connect, "
    SELECT  * 
    FROM    $tb_prices 
    WHERE   $tb_prices.price_brand = '$brand' AND $tb_prices.price_code = '$code' AND $tb_prices.price_validity = (
            SELECT  MAX($tb_prices.price_validity) 
            FROM    $tb_prices 
            WHERE   $tb_prices.price_validity<=DATE_ADD(CURDATE(), INTERVAL +0 DAY) AND $tb_prices.price_code = '$code'
    )
    GROUP BY    $tb_prices.price_code           
    ") ;

}

Probably is not the best and elegant solution but for me works pretty well!

  • Putting a query in a while iteration can lead to preformance issues if you need to iterate many times! I see that the only parameter of the second query is the `price_code`. What about a single query with no while, with an `IN` clause? – Phate01 Dec 06 '17 at 13:52
  • @Phate01, I Know tha this is not the better solution however it works pretty well also tith a large number of reiteration (thousands of items). Could you suggest me a better solution using an `IN` clause? Becouse, as you can read in the main question, I start from a single query (using JOIN) but it doesn't work! – ActiveMedia Dec 06 '17 at 15:33
  • My experience lead to this solution: the scenario is that I do a query, and for each item returning from this query I need to do another query. In your case, for each 'product_code' I need to query to retrieve the prices data. You can create a collection of product codes from the results of the first query. Then do the second query only once, setting 'IN' instead of '=' in the where clause, selecting 'product_code', tb_prices.* and retrieving data back with a while. Like this you access the db only twice instead on 1 + n times where n is the number of the records from the first query – Phate01 Dec 06 '17 at 16:25
  • I understood your explanation and the logic of operation.. but I'm not sure I understood how to put it into practice. Is it too much if I ask you to give me an example? It's a really interesting solution and I would like to understand how to exploit it! Thank you so much! – ActiveMedia Dec 07 '17 at 13:39