0

I have VPS on hetzner. Server is located in Germany.

It has 256GB RAM, 6CPUs (12 threads).

I have a file which since yesterday, is requested about 30 times in one second. File has 2 Select, 2 Update, 2 Insert queries, so I assumed (not sure how this works) from this file server has about 180 requests per second. So right after this requests started, all the websites on the server just started loading poorly. I made this file run just one select query and than die. This didn't help. In WHM load is aboiut 0.02.

I've checked for error logs and there is no max_user_connection or any error there.

I have enabled slow query log and checked log file. there is nothing (I've tested it with select sleep(10) and this query was logged).

This is visit statistics, please bring your attention to may 30th:

enter image description here

Bandwidth stats for last 24 hours:

enter image description here

There are many errors like this in ssl_log (diff IPs of course):

188.121.206.150 - - [30/May/2018:19:50:03 +0200] "-" 408 - "-" "-"

I've been searching web a lot and couldn't find any solution. Could anyone at least tell what should I monitor or where. I have full access to anything there is possible inside the server. Any help is appreciated.

UPDATE 1

I have subdomain: banners.analyticson.com (access allowed for now) and there I have all the images and html5 files that are requested. Take one image for example : https://banners.analyticson.com/img/suy8G1S6RU.jpg

It needs too much time to load. As I noticed, this sub domain has some issue. Script, that I mentioned earlier (with 6 queries) just tries to get one of those banners to the user, so result of that script is to return one banner from banners.analyticson.com.

UPDATE 2

I've checked my script, it is fine. It takes less than 1 second to complete. I've also checked Top command and there is a result. I'm not sure if $MEM value is fine.

enter image description here

temo
  • 612
  • 1
  • 9
  • 25
  • Is your page correctly closing and disposing of any DB connections ASAP after use? – Paddy May 31 '18 at 08:16
  • @Paddy I do have `mysqli_close($conn)` written in my script if you are referring to that. If not please provide some info where can I monitor connections that you are talking about. – temo May 31 '18 at 08:23
  • @Paddy I have just checked `show status where `variable_name` = 'Connections'` and _Connections_ value is 57337 but as I've read it is not simultaneous connections.query: `Show global status like '%Thread%'` seems just fine. Cached threads is 2, Threads connected:3, Threads created: 5, running: 1 – temo May 31 '18 at 08:27

1 Answers1

1

You're going to have to narrow the problem down...

There are multiple potential issues.

First thing to eliminate would be the performance of your new script on a development laptop - I assume you're using PHP, so use the profiling tools to work out what is going on. If it's a database query, you'll see which one by looking at the profiler.

If your PHP script and database queries are fine, the next thing to look at: it sounds like you've hit some bottleneck resource on your infrastructure. In these cases, scripts that run fine as a single request start queueing for the bottleneck resource, and every new request adds to the queue until the whole server starts to crawl. This can be a bit of a puzzle - start with top and keep digging.

Next, I'd look at configuration of Apache to make sure everything is squeaky clean - Apache used to have a default to do a reverse DNS lookup for every request, which slows the server down rather impressively on production. You may also want to look at your SSL configuration - the error you report is linked to a load balancer issue.

If it's not as simple as memory, CPU etc., you're into more esoteric issues. You may need to ramp up a load testing rig so you can experiment without affecting the live site - typically, I do this on a machine as similar to live as possible, using Apache JMeter to generate load, and find the "inflection point". Typically, you see response times increase linearly with the number of concurrent requests, until you hit the bottleneck resource, at which point the response time increases rapidly. As a simple example, if you have 10 database connections available, response time should increase linearly up to 10 concurrent connections, and then become much larger from 11 up.

Knowing where the inflection point is and being able to recreate it allows you to use PHP profiling tools under load. This is a lot of work.

UPDATE

You're using php-cgi; this is easily the most inefficient way of running PHP scripts. Your server is barely breaking a sweat - CPU and memory basically idle. Here's a comparison for how to run PHP; consider changing to mod_php.

Neville Kuyt
  • 29,247
  • 1
  • 37
  • 52
  • Thank you for your answer. I've checked script and it is fine, I checked top and i'm not sure. Could you look at Updated question please? I'm now checking apache configuration. – temo Jun 04 '18 at 12:09
  • I think your UPDATE does it. mod_php (DSO in EA4) helped a lot. After changing it just caches my iframes but this is easily fixable. Thank you very much Neville. – temo Jun 08 '18 at 11:30
  • Not I have another problem with this. When retreiving data through SSL it takes too long to process. HTTP works just fine. I've looked over your suggestions several times and no luck. Could you provide some additional info on how can I resolve this issue? – temo Jun 08 '18 at 18:26
  • Please open a new question, and provide as much information as you can - how long is "too long"? What have you tried? Can you share some SSL URLs we can look at? – Neville Kuyt Jun 11 '18 at 09:57