0

I am using Spark 2.3.1 on Mac, in Java.

I have confidential security info stored in an environment variable. However, as it's confidential, I don't want to expose its value through ps -e nor from http://localhost:4040/environment/.

Is there a way within Spark for me to hide the value please? Or anyway by code seal the value, while not affecting other Spark/Java functions.

mck
  • 40,932
  • 13
  • 35
  • 50
sue
  • 43
  • 4
  • I would use a container engine like docker for spark to keep it from accessing a private environment. You can also change the environment variable in the spark script before it starts. There is a startup script in the config folder – Light Jul 05 '18 at 23:41

1 Answers1

0

You shouldn't use environment variables to store confidential information there are tools like Hashicorp Vault

regarding ps -e - if someone has access to the machine they can echo the environment variable anyway. As for the spark UI - you can secure the access to it see here

Arnon Rotem-Gal-Oz
  • 25,469
  • 3
  • 45
  • 68