4

There are several discussions around the web about some "anonymous" benchmarks released by Litespeed Technologies on http2benchmark.org (etc) allegedly showing that their LS servers are faster than Nginx LEMP stacks (apparently, they knowingly disabled FastCGI caching):

https://hostedtalk.net/t/litespeed-servers-publishing-fake-benchmarks/4431

https://www.reddit.com/r/selfhosted/comments/f06vse/litespeed_servers_seem_like_a_marketing_scam_are/

https://community.centminmod.com/threads/is-lsapi-litespeed-faster-than-php-fpm-nginx-etc-or-not.19153/

My question is three-fold:

  1. As far as LSAPI on Litespeed vs PHP-FPM on Nginx is concerned, what would be a "fair" way to compare performance re: processing PHP queries since the environments are different?
  2. Besides the PHP handler, is there any other part of the Litespeed stack that can be accurately benchmarked at the application-level (e.g. LS Cache) instead of the stack?
  3. Have any unbiased third parties compared these two stacks in an accurate fashion?
Jesse Nickles
  • 1,435
  • 1
  • 17
  • 25
  • This question isn't exactly a programming question so it could be considered off-topic for Stack Overflow. I'm not sure it's a good fit for ServerFault either, though. The part asking for off-site resources (3rd-party benchmarks) is almost certainly off-topic on SO and other SE sites. But parts of this question (about benchmark methodology) could be interesting and relevant; *maybe* we should make an exception to the rules and let it stand. (Or not, and people should go talk about it elsewhere like reddit.) I'm not going to vote-to-close, but don't be surprised if other people do. – Peter Cordes Mar 16 '20 at 06:24
  • @PeterCordes I agree, there are many similar benchmarking questions as well. I think maybe SE needs to loosen its rules on "opinion" since they now also encourage authors to answer their own questions as well, in a bid to take on Quora better. – Jesse Nickles Mar 16 '20 at 06:27
  • Part of the problem with questions that simply ask for benchmark *results* is that they don't age well. SO's primary purpose is a repository of useful Q&As that you can search. The methodology part of your question can stand the test of time, and any analysis of *why* one specific benchmark isn't representative can bring up useful things to look for. Questions about *how* to benchmark are fine. – Peter Cordes Mar 16 '20 at 06:35
  • Even with the above discussion, I fail to see the [tag:apache] tag's relevance – Olaf Kock Mar 16 '20 at 06:46
  • @OlafKock There is a lot of overlap between Litespeed and Apache, and Litespeed markets itself as a drop-in replacement for Apache servers. – Jesse Nickles Mar 16 '20 at 08:03

1 Answers1

1

I'd rather post this as a comment but it's pretty full already up there.

This benchmark routine by Ike Devolder is probably one of the most fair and detailed that I've come across in the past few years comparing Litespeed and Nginx:

https://blog.herecura.eu/blog/2020-06-16-openlitespeed-vs-apache-vs-nginx/

Some of the confusion is re: Litespeed vs. OpenLitespeed, but for this purpose it doesn't really matter since the biggest difference with Nginx is LSAPI, the PHP processor, rather than using PHP-FPM, and since OLS supports the same server-level caching that Litespeed does...

One of the best things about Ike's review is that he doesn't definitively say one is "better" or "faster" and instead talks about different components of either stack, while monitoring resources.

The TLDR for 2021 is probably something like this:

  1. Both Nginx and Litespeed serve static assets extremely well
  2. Litespeed as a stack scales higher than Nginx currently, partially due to HTTP/3 support... but 99% of websites probably will not notice this since we are talking about insane traffic levels, which almost never reach the origin server these days due to things like CDNs or remote MySQL servers, and since HTTP/3 is not fully integrated into the internet yet... in fact, even in Ike's own tests he seems to have tested both internally (not over i.e. UDP) which further reinforces this point since HTTP/3 is UDP-only. However, regardless of protocol, Litespeed does appear to scale to a higher level than Nginx... keep in mind, this is not the same as saying that LS delivers requests "faster" than Nginx, because real world requests at a typical traffic level are probably indistinguishable in terms of speed/response times. It does mean that i.e. a shared hosting environment with 500 cPanel websites would probably scale better using Litespeed... so it makes sense why that seems to be Litespeed's main target.
  3. Litespeed appears to use more RAM memory than Nginx in general
  4. Nginx appears to use a lot more CPU than Litespeed at high traffic levels
  5. TCP (not Unix socks, the Litespeed default) performed a lot better, and this is true for Nginx as well, since it avoids disk I/O... so, using default Litespeed settings is still a bad idea

Anyway, best not to get too detailed, since software changes all the time really. The next version of Nginx will probably support HTTP/3, among other features. I do wish that the Litespeed guys would let the unique quality of their software speak for itself, and chill out on the aggressive (inaccurate) claims being made... but most of their web hosting partners probably benefit from that hype, which is maybe why not many people are questioning the claims being made...

Jesse Nickles
  • 1,435
  • 1
  • 17
  • 25