2

I am running Prestashop 1.6 powered e-shop on VPS with 4GB RAM, 2CPUs, 60GB SSD characteristics. Currently I have about 20000 products in my shop and I am having problems with site loading because mysql queries run long. And when I run htop to analyze processes, I see that mysql consumes 100% of both CPUs. This is the output from mysqltuner:

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1h 29m 9s (241K q [45.109 qps], 319 conn, TX: 318M, RX: 126M)
[--] Reads / Writes: 78% / 22%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[OK] Maximum possible memory usage: 597.8M (15% of installed RAM)
[OK] Slow queries: 0% (8/241K)
[OK] Highest usage of available connections: 2% (4/151)
[OK] Key buffer size / total MyISAM indexes: 16.0M/44.9M
[OK] Key buffer hit rate: 99.6% (36M cached / 133K reads)
[OK] Query cache efficiency: 49.6% (101K cached / 205K selects)
[!!] Query cache prunes per day: 1386761
[OK] Sorts requiring temporary tables: 0% (1 temp sorts / 7K sorts)
[!!] Joins performed without indexes: 78
[OK] Temporary tables created on disk: 8% (846 on disk / 9K total)
[OK] Thread cache hit rate: 98% (4 created / 319 connections)
[!!] Table cache hit rate: 10% (340 open / 3K opened)
[OK] Open file limit used: 62% (643/1K)
[OK] Table locks acquired immediately: 100% (239K immediate / 239K locks)

-------- Recommendations -----------------------------------------------------
General recommendations:
    Add skip-innodb to MySQL configuration to disable InnoDB
    MySQL started within last 24 hours - recommendations may be inaccurate
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
    query_cache_size (> 16M)
    join_buffer_size (> 128.0K, or always use indexes with joins)
    table_cache (> 400)

Please suggest any optimization methods.

EDIT:

Slow query output log is here.

torayeff
  • 9,296
  • 19
  • 69
  • 103
  • So basically you're seeking professional tuning help for free, plus you don't know where to start optimizing? – N.B. Oct 21 '14 at 09:43
  • How about getting a slow query log and posting the actual culprit SQL queries that are killing performance. A bunch of stats vs not seeing poorly written queries and known indexes might really be where things are choking. I don't know if I would agree with the slow queries 0% percent as indicated. Notice your JOINS WITHOUT INDEXES = 78 – DRapp Oct 21 '14 at 12:01
  • @DRapp here is the output of slow query log http://pastie.org/9664946 – torayeff Oct 21 '14 at 12:22
  • 1
    @N.B. If I knew I would not have asked. Be it professional tuning or not I have a right to learn. – torayeff Oct 21 '14 at 12:24
  • Of course you have the right to learn, I would never question that. However, you posted output of a program and then what, you expect spoon feeding? For example, the program tells you this: `Enable the slow query log to troubleshoot bad queries`. Google > "Enable the slow query log to troubleshoot bad queries" > if in trouble - ask here. This way, it appears like you want everything done for you, because the question you asked is without sufficient info to help you. It looks like you want a complete solution instead of clues on where to approach the problem. – N.B. Oct 21 '14 at 12:33
  • @N.B. OK I do not want to argue. If you can help. – torayeff Oct 21 '14 at 12:36
  • Start by enabling slow query log. It will log queries that take too much time to complete. Once you have several of those, and if they are SELECT type - add `EXPLAIN` before `SELECT` to see the execution plan. You do this from MySQL terminal or PHPMyAdmin (or any other MySQL GUI). It will tell you what MySQL is trying to do internally to execute the query. Once you have that, post the queries along with table structure here for further analysis. – N.B. Oct 21 '14 at 12:57
  • @N.B This query which I pasted here http://pastie.org/9665033 takes about 12 seconds to execute only to display 8 records. – torayeff Oct 21 '14 at 13:02
  • Can I somehow cache queries? – torayeff Oct 21 '14 at 13:08
  • The query itself is not useful alone. We need to see execution plan. Add `EXPLAIN` before select like this: `EXPLAIN SELECT` and post that back. – N.B. Oct 21 '14 at 13:09
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/63410/discussion-between-torayeff-and-n-b). – torayeff Oct 21 '14 at 13:10
  • explain select ouptut is here http://pastie.org/9665152 – torayeff Oct 21 '14 at 13:47
  • I think this question will help you http://stackoverflow.com/questions/26483174/optimize-mysql-for-prestashop-1-6 – HMagdy Feb 19 '15 at 10:59

2 Answers2

0

Here is the second query from your log, looking into others. To help on this, I slightly restructured to get rid of the subquery and turned into an INNER JOIN. Also, to help others not knowing your tables, always qualify your field references as table.column or alias.column such as to know where your level_depth, nleft, nright are coming from. It may help with indexes.

FOR indexes, I would have the following per table.

ps_category, index ON( active, id_category, id_lang, nleft, nright, level_depth )
ps_category_lang, index ON( id_category, id_shop, id_lang )
ps_category_shop, index ON( id_category, id_shop )  
ps_category_group, index ON( id_category, id_group )
ps_lang, index ON( id_lang, active )

slightly rewritten query

SELECT 
      c.id_parent, 
      c.id_category, 
      cl.name, 
      cl.description, 
      cl.link_rewrite
   FROM 
      ps_category c
         INNER JOIN ps_category_lang cl 
            ON c.id_category = cl.id_category 
            AND cl.id_shop = 1
            AND cl.id_lang = 2 
         INNER JOIN ps_category_shop cs
            ON c.id_category = cs.id_category 
            AND cs.id_shop = 1
         INNER JOIN ps_category_group psg
            ON c.id_category = psg.id_category
            AND psg.id_group = 1
   WHERE 
      (  c.active = 1 
      OR c.id_category = 2)
      AND c.id_category != 1
      AND level_depth <= 7
      AND nleft >= 350 AND nright <= 351
   ORDER BY 
      level_depth ASC, 
      cs.position ASC;

Third query. Notice in this case, some of the key elements that are specific are moved to the top, hence I have them in the key index too, THEN the rest in the where clause (mostly readability adjustments, but helps to see "specific" elements for index advantages )

SELECT 
      c.id_category, 
      cl.name, 
      cl.link_rewrite
   FROM 
      ps_category c
         LEFT JOIN ps_category_lang cl 
            ON c.id_category = cl.id_category 
            AND cl.id_shop = 1
         INNER JOIN ps_category_shop category_shop
            ON c.id_category = category_shop.id_category 
            AND category_shop.id_shop = 1
   WHERE 
          c.active = 1
      AND cl.id_lang = 2
      AND c.nleft between 2 and 350
      AND c.nright between 351 and 625
      AND c.level_depth > 1
   ORDER BY 
      c.level_depth ASC;

The next query you need to confirm what you really wanted... You have a left-join to the language table, but then add "AND l.active = 1" in the where clause which in-fact would turn it into an INNER JOIN. If you want it truly to a LEFT-JOIN, move the l.active to the join part such as I adjusted here

SELECT 
      l.id_lang, 
      c.link_rewrite
   FROM 
      ps_category_lang AS c
         LEFT JOIN ps_lang AS l 
            ON c.id_lang = l.id_lang
            AND l.active = 1
   WHERE 
      c.id_category = 324

Hopefully these indexes and sample clarification / readability of queries might help improve your logs. If still problems with others, post as needed.

REVISION FOR FIRST QUERY IN SLOW LOG REPORT

In the first query, it appears you are looking for things out of a specific product shop. However, you are starting the query with the product category and left-joining to products, then to product shops. Since these products will ultimately be associated with a specific category via the where, it creates an INNER JOIN anyhow. I would slightly restructure as below

I would have a single index on ps_product_shop ON ( id_shop, active, visibility, id_product, id_category_default )

SELECT 
      p.*, 
      ps.*, 
      stock.out_of_stock, 
      IFNULL(stock.quantity, 0) as quantity, 
      MAX(pas.id_product_attribute) id_product_attribute, 
      pas.minimal_quantity AS product_attribute_minimal_quantity, 
      pl.description, 
      pl.description_short, 
      pl.available_now,
      pl.available_later, 
      pl.link_rewrite, 
      pl.meta_description, 
      pl.meta_keywords, 
      pl.meta_title, 
      pl.name, 
      MAX(image_shop.id_image) id_image,
      il.legend, 
      m.name AS manufacturer_name, 
      cl.name AS category_default,
      DATEDIFF(ps.`date_add`, DATE_SUB(NOW(),INTERVAL 20 DAY)) > 0 AS new, 
      ps.price AS orderprice
   FROM 
      ( select @thisLanguage := 1 ) sqlvars,
      ps_product_shop ps
         INNER JOIN ps_product p
            ON ps.id_product = p.id_product

            INNER JOIN ps_category_product cp
               ON id_product = cp.id_product
               AND cp.id_category = 2 

            LEFT JOIN ps_product_attribute pa
               ON p.id_product = pa.id_product
               LEFT JOIN ps_product_attribute_shop pas
                  ON pa.id_product_attribute = pas.id_product_attribute
                  AND ps.id_shop = pas.id_shop
                  AND pas.default_on = 1

            LEFT JOIN ps_stock_available stock
               ON p.id_product = stock.id_product 
               AND ps.id_shop = stock.id_shop
               AND stock.id_shop_group = 0
               AND stock.id_product_attribute = IFNULL(pas.id_product_attribute, 0)

            LEFT JOIN ps_product_lang pl
               ON p.id_product = pl.id_product
               AND ps.id_shop = pl.id_shop 
               AND pl.id_lang = @thisLanguage

            LEFT JOIN ps_image i
               ON p.id_product = i.id_product
               LEFT JOIN ps_image_shop image_shop
                  ON i.id_image = image_shop.id_image 
                  AND ps.id_shop = image_shop.id_shop
                  AND image_shop.cover = 1
                  LEFT JOIN ps_image_lang il
                     ON image_shop.id_image = il.id_image
                     AND il.id_lang = @thisLanguage

            LEFT JOIN ps_manufacturer m
               ON p.id_manufacturer = m.id_manufacturer

         LEFT JOIN ps_category_lang cl
            ON ps.id_category_default = cl.id_category
            AND cl.id_shop = ps.id_shop
            AND cl.id_lang = @thisLanguage
   WHERE 
          ps.id_shop = 1
      AND ps.active = 1 
      AND ps.visibility IN ("both", "catalog") 
   GROUP BY 
      ps.id_product 
   ORDER BY 
      cp.position ASC
   LIMIT 
      0,8;

As for creating the multiple field index as a single index, example:

CREATE INDEX act_id_lang ON ps_category( active, id_category, id_lang, nleft, nright, level_depth );

give a simple index name... on whatever table... with ALL the keys as they will be effective in the query.

DRapp
  • 47,638
  • 12
  • 72
  • 142
  • For the second query I have Query_time: 0.001116. Why I need to optimize it? The first query is long Query_time: 11.635315 – torayeff Oct 21 '14 at 13:50
  • @torayeff, I will look back at the first query, but the indexes themselves should help, and the queries as written show me / help me identify what indexes might be best to help across-the-board. I'll get back to the first query soon. – DRapp Oct 21 '14 at 14:23
  • @torayeff, that may be... but are they individual indexes, or a COMPOUND index of all the fields as I have listed as a SINGLE index... Completely different results when querying. – DRapp Oct 21 '14 at 14:31
  • can I give you access to my phpmyadmin? – torayeff Oct 21 '14 at 14:33
  • I meant how can I give you access to my phpmyadmin? Where I can send you? – torayeff Oct 21 '14 at 14:34
  • @torayeff, can't help remotely now, but added to answer for the first query restructured and sample syntax to create a multi-column index to help. – DRapp Oct 21 '14 at 15:02
  • 1
    This is not an index issue. No satisfying criteria exists to use any index, plus the cardinailty is obviously low. It's a simple configuration issue, he's using MyISAM engine and the database has to go through 20k records. Sorting on disk, even if SSD, is slower than utilizing RAM. It's also a web shop, running with no transactional support. There are many errors involved here, it's a hot potato. I suggest hiring an expert, it's much cheaper in the long run and you obviously are not here to learn. – N.B. Oct 21 '14 at 15:24
0

You should use mysqltuner after a longer usage time. This should remove the following message : MySQL started within last 24 hours - recommendations may be inaccurate

jmrenouard
  • 104
  • 3