1

I'm having difficulties identifying what causes my website to load extremely slow, I have found something but google archives don't provide the right answer or even explanation. In my raw-access logs I found multiple records about different robots accessing my website, here's an example:

202.46.53.40 - - [31/Dec/2016:03:30:51 +0100] "GET /en/home/184-2016-hyperlite-motive-wakeboard.html HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.54.27 - - [31/Dec/2016:03:30:52 +0100] "GET /en/home/184-2016-hyperlite-motive-wakeboard.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.56.210 - - [31/Dec/2016:03:30:53 +0100] "GET /en/home/184-2016-hyperlite-motive-wakeboard.html HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.56.114 - - [31/Dec/2016:03:30:54 +0100] "GET /en/wakeboards/184-2016-hyperlite-motive-wakeboard.html HTTP/1.1" 200 140041 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
180.76.15.154 - - [31/Dec/2016:03:31:26 +0100] "GET /en/26-sup HTTP/1.1" 406 73864 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
157.55.39.40 - - [31/Dec/2016:03:31:50 +0100] "GET /en/helmets/57-2015-mystic-mk8-helmet-mint.html HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.39.40 - - [31/Dec/2016:03:31:55 +0100] "GET /en/helmets/57-2015-mystic-mk8-helmet-mint.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
77.75.77.95 - - [31/Dec/2016:03:34:03 +0100] "GET /robots.txt HTTP/1.1" 404 57839 "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
77.75.77.95 - - [31/Dec/2016:03:34:05 +0100] "GET /en/31-bags HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
163.172.66.143 - - [31/Dec/2016:03:43:36 +0100] "GET /en/13-rokavice HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
202.46.54.134 - - [31/Dec/2016:04:04:20 +0100] "GET /en/accessories/169-plavutke-pro-ii.html HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.54.102 - - [31/Dec/2016:04:04:21 +0100] "GET /en/accessories/169-plavutke-pro-ii.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
202.46.48.140 - - [31/Dec/2016:04:04:22 +0100] "GET /en/accessories/169-plavutke-pro-ii.html HTTP/1.1" 200 110602 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
180.76.15.10 - - [31/Dec/2016:04:04:55 +0100] "GET /en/56-kiteboarding-gear HTTP/1.1" 406 62988 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
66.249.76.47 - - [31/Dec/2016:04:25:33 +0100] "GET /380/komplet-oceanrodeo-razor-fst8-advenced-performance-kite.jpg HTTP/1.1" 200 126044 "-" "Googlebot-Image/1.0"
112.210.233.49 - - [31/Dec/2016:04:29:17 +0100] "POST /modules/sendtoafriend/sendtoafriend_ajax.php?rand=1472104141118 HTTP/1.1" 500 - "https://proadrenalin.si/modules/sendtoafriend/sendtoafriend_ajax.php?rand=1472104141118" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
66.249.76.78 - - [31/Dec/2016:04:33:09 +0100] "POST /modules/leocustomajax/leoajax.php?rand=1482019200024 HTTP/1.1" 200 14 "https://www.proadrenalin.si/en/20-wakeboards?p=3" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Is it possible that this visits are causing the problem with slow page load ? For 31st December i have 1342 requests, 1st Jan. 1222, 2nd Jan - 2374 requests, 4th Jan - 2391... This goes on every day. Webshop is run by Prestashop, and as far as I've been inspecting the platform is not causing any problems which would result in slow page load. Most of modules are disabled, removed, only needed (enabled) ones are on server, cashing is on, recompile when something changes..

Any tips, links to read, possible solutions...would be very useful because currently I'm living in nightmare..

Blue
  • 261
  • 1
  • 3
  • 14
  • if you put the store on maintenance, and add only your ip, is it still slow? if it is, it's not because of the bots crawling. also you can deny bots from crawling your pages, or certain pages/sections. – sadlyblue Jan 27 '17 at 17:49
  • 2
    I voted to close this question because it is not a programming question and it is off-topic on Stack Overflow. Non-programming questions about your website should be asked on [webmasters.se]. In the future, please ask questions like this there. – Stephen Ostermiller Jan 07 '22 at 11:51

2 Answers2

2

You can find the IP patterns of the robots hitting your store and then block those IPs using the .htaccess file.

Visit following URL for more details on this:

How to Block an IP address range using the .htaccess file

Community
  • 1
  • 1
Knowband Plugins
  • 1,297
  • 1
  • 7
  • 6
  • 2
    I think blocking IP's is not solution because webstore is shipping worldwide and I think at some point I may be also blocking real customers when IP's change, and, from reading log file, IP's from bots are changing completly. You think this would be good idea ? – Blue Feb 03 '17 at 11:00
0

I have the same problem and currently trying a solution to block that bot via robots.txt like this:

User-agent: SeznamBot
Disallow: /

Taken from official source https://napoveda.seznam.cz/en/full-text-search/crawling-control/

Artem Okhrei
  • 31
  • 2