0

How to set a global variable in Spark. ?
I have few files in spark in which I need to pass a parameters. But I want to set that variable in one file( file name - kafkaCom)

We have topic named as "test" in one environment of kafka. I have two more instances. So If we submit a job from first environment, lets suppose TEST1. So I want to call test_one. If job submitted from TEST2, Then it would call test_two. If job submitted from TEST3, Then it would call test_three etc.

So I want that keyword which is appending at last in test_ as per of job submission environment.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
dataeng
  • 11
  • 4
  • 1
    Your question (at least as far as it's understandable from the description) is not related to Spark, you need to access some variable from some other file. And that variable is in some class, or it's static (in some public object). If it's the first case, you can either extend the class, or create an object of that class, etc,. If it's static, just import it. – AminMal Jun 15 '22 at 08:13
  • Please show a [mcve] of one job so we can understand what you're trying to accomplish – OneCricketeer Jun 15 '22 at 14:18

1 Answers1

0

submit a job from first environment, lets suppose TEST1. So I want to call test_one. If job submitted from TEST2, Then it would call test_two. If job submitted from TEST3, Then it would call test_three etc

In first environment,

spark-submit ... --class your.code.SparkApp code.jar one

In second environment,

spark-submit ... --class your.code.SparkApp code.jar two

And so on.

Then access the argument in your main function and use string interpolation

object SparkApp extends App{
    def main(args: Array[String]): Unit {
      val environment = args(0)
      
      val spark = SparkSession.builder.appName(s"test_${environment}").getOrCreate()
    }
}
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245