1

I'm using spring-boot and spring-boot-yarn to submit yarn applications to a cluster.

My use-case is close to the one described in this tutorial https://github.com/spring-guides/gs-yarn-basic.

The only difference is that my 'client' is supposed to be a web application and submit the yarn jobs when web requests are made.

The problem I have is that web requests to the 'client' web-application provide parameters I need to pass down to the yarn job.

In the above tutorial parameters are passed as command line arguments to to the appmaster / container specified in application.yml. In my case this approach does not work since I have a different set of parameters for each yarn job.

Is there a way to pass dynamic parameters to yarn jobs without hard-coding them in application.yml?

Victor Iacoban
  • 13
  • 4

1 Answers1

1

Original idea was to prevent "rogue" users or applications to pass properties which would then automatically end up in a command-line options potentially making harm within a hadoop cluster.

It's worth to check my answer in Spring Boot Yarn - Passing Command line arguments if this is what you want.

Having said that, you are not a first person to ask this or "complain" that it is too difficult or unclear how to do it. We're going to make this much easier with future releases mostly because it just seem to be what users want to do.

Community
  • 1
  • 1
Janne Valkealahti
  • 2,552
  • 1
  • 14
  • 12
  • Thanks Janne, the above link does not satisfy my use-case but I still found your answer useful. It seemed pretty obvious that there is no support in API to pass down parameters to yarn apps (unless it's a one time, fixed value parameter as in command line example) as a result I had to revisit my architecture and and switch to a command line app vs web app. – Victor Iacoban Apr 24 '15 at 15:12
  • Right ok now I "got" it what you meant. There is sort of an API which can be used programmatically but it's not the easiest one to use and that's why I mentioned that we need to make it easier. YarnSubmitCommand.java from spring-yarn-boot-cli is using YarnSubmitApplication.java from spring-yarn-boot to do the submit logic. There's other command/app classes for other supported logic. Every command is an instance of app context run by boot. It was done like this to support all nice boot features around how app properties can be used(env, props, yml, etc). – Janne Valkealahti Apr 25 '15 at 07:22