1

I'm using Netflix/Archaius library for configuration management of scala/spark job. currently my property files are in the java properties format like 

com.comp.appname.solrconfig.collectionname=testcollection
com.comp.appname.solrconfig.url=http://someip:8983/solr
com.comp.appname.solrconfig.zkhost=someip:2181
com.comp.appname.sparkconfig.executors=20
com.comp.appname.sparkconfig.executormb=200

would like to convert it to yaml format (which is more simple and readable)

solrconfig:
   collectionName: testcollection
   url: "http://someip:8983/solr"
   zkhost: "someip:2181"

sparkConfig: 
   executors: 20
   executormb: 200

My need is to use Netflix/archaius library for initiating a scala spark job.

venBigData
  • 600
  • 1
  • 8
  • 23
  • Are you trying to convert the first format into the second one using Spark? What have you tried so far? – Oli May 10 '19 at 12:34
  • No. I'm currently using the first format. My current configuration file, config.properties is in the first format. I would need to change that to a yaml format (second format). – venBigData May 10 '19 at 15:25
  • In order to use YAML, I tried using snakeyaml with spark-shell and noticed the error as posted in https://stackoverflow.com/questions/38002883/snakeyaml-and-spark-results-in-an-inability-to-construct-objects/40688619#40688619. But that wouldn't solve my problem. I would need to have the Netflix/Archiaus API parse a yaml properties. – venBigData May 10 '19 at 15:32

0 Answers0