4

We are benchmarking spark with alluxio and presto with alluxio. For evaluating the performance we took 5 different queries (with some joins, group by and sort) and ran this on a dataset 650GB in orc.

Spark execution environment is setup in such a way that we have a ever running spark context and we are submitting queries using REST api (Jetty server). We are not considering first batch execution time for this load test as its taking little more time because of task deserialization and all.

What we observed while evaluating is that when we ran individual queries or even all these 5 queries executed concurrently, spark is performing very well compared to presto and is finishing all the execution in half the time than of presto.

But for actual load test, we executed 10 batches (one batch is this 5 queries submitted at the same time) with a batch interval of 60 sec. At this point presto is performing a lot better than spark. Presto finished all job in ~11 mins and spark is taking ~20 mins to complete all the task.

We tried different configurations to improve spark concurrency like

  • Using 20 pools with equal resource allocation and submitting jobs in a round robin fashion.
  • Tried using one FAIR pool and submitted all jobs to this default pool and let spark decide on resource allocations
  • Tuning some spark properties like spark.locality.wait and some other memory related spark properties.
  • All tasks are NODE_LOCAL (we replicated data in alluxio to make this happen)
  • Also tried playing arround with executor memory allocation, like tried with 35 small executors (5 cores, 30G) and also tried with (60core, 200G) executors.

But all are resulting in same execution time. We used dstat on all the workers to see what was happening when spark was executing task and we could see no or minimal IO or network activity . And CPU was alway at 95%+ (Looks like its bounded on CPU) . (Saw almost similar dstat out with presto)

Can someone suggest me something which we can try to achieve similar or better results than presto?

And any explanation why presto is performing well with concurrency than spark ? We observed that presto's 1st batch is taking more time than the succeeding batches . Is presto cacheing some data in memory which spark is missing ? Or presto's resource management/ execution plan is better than spark ?

Note: Both clusters are running with same hardware configuration

Piotr Leniartek
  • 1,177
  • 2
  • 14
  • 33
Rijo Joseph
  • 1,375
  • 3
  • 17
  • 33
  • 2
    For the first test with Presto, your cluster is likely misconfigured. Presto is very good at single query performance, and should be available to use all of the available resources. Normally, this is caused by not having enough data (splits), and to get full parallelism. When you run multiple queries, you can utilize all resources. – Dain Sundstrom May 02 '18 at 17:37
  • Which versions of Presto and Spark are you using? This can make a big difference. Also, Spark tends to process Parquet formatted data more efficiently than ORC. Spark only recently (2.3) got support for a _vectorized_ ORC reader. https://issues.apache.org/jira/browse/SPARK-16060 and here is some more information detailing the performance benefits of the new reader https://www.slideshare.net/Hadoop_Summit/performance-update-when-apache-orc-met-apache-spark-81023199 – Jon May 02 '18 at 17:56
  • 2
    Answering one of your questions -- presto doesn't cache data in memory (unless you use some custom connector that would do this). Anyway -- you compare Presto out-of-the-box performance with Spark cluster you used your time and expertise to tune. Even if you eventually get Spark running on par or faster, it sill won't be a fair comparison. You should use time *and* expertise to configure Presto optimally as well, otherwise this is apples vs oranges. – Piotr Findeisen May 03 '18 at 07:15
  • @Jon spark version is 2.3.0 and presto 0.194 – Rijo Joseph May 04 '18 at 04:52
  • 2
    @PiotrFindeisen Actually we have spend enough time on presto tuning as well. Initially presto was using only 70% of the cpu and we have tuned it to make it nearly 95%cpu – Rijo Joseph May 04 '18 at 04:54
  • As for me it's expected result. Spark Sql (Thrift server) don't handle multitenant behave well. that what they actually declare in their doc. – Grigoriev Nick May 11 '18 at 13:03

0 Answers0