How to handle environmental variable in spark scala

Question

I am very new to spark. Please help me with below query:

Suppose I have different file path in dev and prod and i do not want to hardcode the same in my spark scala code rather I want to use variable in my code so my doubt is that where I define these variables for dev and prod and how to access them in my code.

Vikramsinh Shinde · Answer 1 · 2022-06-29T18:49:28.903

0

Here are a few options:

Pass variables as command line parameters to your Spark application
Use the property/config file to keep those variables
Create environmental variables for these variables

For implementation, follow these articles.

edited Jun 29 '22 at 18:49

answered Jun 29 '22 at 03:45

Vikramsinh Shinde

2,742
2
23
29

@vikramsinhThanks for the response, it would be very helpful if you share how to set env variable for the variables and use them in the application or share some reference – Ankit Gupta Jun 29 '22 at 17:26
Updated the answer to include a few useful links – Vikramsinh Shinde Jun 29 '22 at 18:50

How to handle environmental variable in spark scala

1 Answers1