Spark properties file read

Question

I tried to read a properties file in spark where my file location is available while run the job getting below error code is

object runEmpJob {    
  def main(args: Array[String]): Unit = {

    println("starting emp job")
    val props = ConfigFactory.load()
    val envProps = props.getConfig("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")

    System.setProperty("hadoop.home.directory", "D:\\SHARED\\winutils-master\\hadoop-2.6.3\\bin")

    val spark = SparkSession.builder().
         appName("emp dept operation").
         master(envProps.getString("Dev.executionMode")).
         getOrCreate()
    val empObj = new EmpOperation

    empObj.runEmpOperation(spark, "String", fileType = "csv")

    val inPutPath = args(1)
    val outPutPath = args(2)    
  }
}


getting error:

Exception in thread "main" com.typesafe.config.ConfigException$BadPath: path parameter: Invalid path C:\Users\mmishra092815\IdeaProjects\use_case_1\src\main\Resource\filepath.properties': Token not allowed in path expression: ':' (you can double-quote this token if you really want it here)
at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:155)
at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:74)
at com.typesafe.config.impl.PathParser.parsePath(PathParser.java:61)
at com.typesafe.config.impl.Path.newPath(Path.java:230)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:192)
at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274) at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41) at executor.runEmpJob$.main(runEmpJob.scala:12)
at executor.runEmpJob.main(runEmpJob.scala)
Process finished with exit code 1

score 1 · Answer 1 · answered Jan 03 '20 at 12:18

Loading happens in ConfigFactory.load(). If you want to load configuration from specific file, pass it like:

val props = ConfigFactory.load("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")

As described in API documentation, getConfig method does not load configuration from file - it returns a Config object for given config path (not filesystem path!)

Spark properties file read

1 Answers1