1

Problem explanation

Suppose you have Spark cluster with Standalone manager, where jobs are scheduled through SparkSession created at client app. Client app runs on JVM. And you have to launch each job with different configs for the sake of performance, see Job types example below.

The problem is you can't create two sessions from single JVM.

So how you gonna launch multiple Spark jobs with different session configs simultaneously?

By different session configs I mean:

  • spark.executor.cores
  • spark.executor.memory
  • spark.kryoserializer.buffer.max
  • spark.scheduler.pool
  • etc

My thoughts

Possible ways to solve the problem:

  1. Set different session configs for each Spark job within the same SparkSession. Is it possible?
  2. Launch another JVM just to start another SparkSession, something that I could call Spark session service. But you never knew how many jobs with different configs you gonna launch in future simultaneously. At the moment - I need only 2-3 different configs at a time. It's may be enough but not flexible.
  3. Make global session with the same configs for all kinds of jobs. But this approach is a bottom from perspective of performance.
  4. Use Spark only for heavy jobs, and run all quick search tasks outside Spark. But that's a mess, since you need to keep another solution (like Hazelcast) in parallel with Spark, and split resources between them. Moreover, that brings extra complexity for all: deployment, support etc.

Job types example

  1. Dump huge database task. It's CPU low but IO intensive long running task. So you may want to launch as many executors as you can with low memory and cores per executor.
  2. Heavy handle-dump-results task. It's CPU intensive so you gonna launch one executor per cluster machine, with maximum CPU and cores.
  3. Quick retrieve data task, which requires one executor per machine and minimal resources.
  4. Something in a middle between 1-2 and 3, where a job should take a half of cluster resources.
  5. etc.
Community
  • 1
  • 1
VB_
  • 45,112
  • 42
  • 145
  • 293
  • It can be achieved but requires quite a lot analyze time. – FaigB Mar 09 '17 at 15:06
  • @FaigB we have already spent quite a lot of analyze time, and haven't found acceptable solution. Could you please share at least minimal consideration of what you gonna analyze? – VB_ Mar 09 '17 at 15:08
  • You can use customized manager which will create process with required spark configs and launch your preferred type of app as spark job – FaigB Mar 09 '17 at 15:31
  • @FaigB please more details. Customized manager of what? Do you mean extend Spark functionality? Or there are already solutions for that? – VB_ Mar 09 '17 at 15:44
  • 2
    Per example if you are using Yarn. With help of yarn manager you can easily manage queue with specific resource allocation which will limit job resource consumption. Also restrict simultaneously run in same queue. Just need in launch config to provide which queue will be used. – FaigB Mar 09 '17 at 15:58
  • @FaigB could I have multiple queues run in parallel with YARN? – VB_ Mar 09 '17 at 17:09
  • Yes it depends how you are going to allocate cluster resources among them. – FaigB Mar 09 '17 at 20:51
  • @FaigB is it possible with Standalone manager? I think switching to YARN may take pretty much time – VB_ Mar 10 '17 at 15:00
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/137768/discussion-between-faigb-and-volodymyr-bakhmatiuk). – FaigB Mar 10 '17 at 15:18

1 Answers1

1

Spark standalone uses a simple FIFO scheduler for applications. By default, each application uses all the available nodes in the cluster. The number of nodes can be limited per application, per user, or globally. Other resources, such as memory, cpus, etc. can be controlled via the application’s SparkConf object.

Apache Mesos has a master and slave processes. The master makes offers of resources to the application (called a framework in Apache Mesos) which either accepts the offer or not. Thus, claiming available resources and running jobs is determined by the application itself. Apache Mesos allows fine-grained control of the resources in a system such as cpus, memory, disks, and ports. Apache Mesos also offers course-grained control control of resources where Spark allocates a fixed number of CPUs to each executor in advance which are not released until the application exits. Note that in the same cluster, some applications can be set to use fine-grained control while others are set to use course-grained control.

Apache Hadoop YARN has a ResourceManager with two parts, a Scheduler, and an ApplicationsManager. The Scheduler is a pluggable component. Two implementations are provided, a CapacityScheduler, useful in a cluster shared by more than one organization, and the FairScheduler, which ensures all applications, on average, get an equal number of resources. Both schedulers assign applications to a queues and each queue gets resources that are shared equally between them. Within a queue, resources are shared between the applications. The ApplicationsManager is responsible for accepting job submissions and starting the application specific ApplicationsMaster. In this case, the ApplicationsMaster is the Spark application. In the Spark application, resources are specified in the application’s SparkConf object.

For your case just with standalone it is not possible may be there can be some premise solutions but I haven't faced

FaigB
  • 2,271
  • 1
  • 13
  • 22
  • could you please look at this question http://stackoverflow.com/questions/42860835/sparkcontext-setlocalproperties?noredirect=1#comment73002450_42860835? It's a continuetion of the current one – VB_ Mar 22 '17 at 14:06
  • @FraigB First of all - thank u for your answers! Could you pls explain me one thing about YARN? The key point here is single Spark session (single JVM). Is it possible to solve the problem with YARN within single Spark session/application? Could you pls take a look at http://stackoverflow.com/questions/43940349/dynamic-resource-allocation-cores-ram-for-parallel-jobs-within-single-spark-s – VB_ May 12 '17 at 14:37
  • also Apache Mesos deprecated fine-grained mode http://spark.apache.org/docs/latest/running-on-mesos.html#fine-grained-deprecated. Could I solve the problem with coarse-grained mode and dynamic allocation? – VB_ May 15 '17 at 11:20
  • This answer about standalone may seem to make sense but it is actually incorrect (see my answer) – WestCoastProjects Sep 08 '18 at 23:16