3

I am new to Spring cloud data flow. Trying to figure out what it mean by registering, creating a task on spring cloud data flow server vs running a spring class with annotation @EnableTask Any clarification to understand what these two are, will be helpful!

mjs
  • 139
  • 8

1 Answers1

3

With Spring Cloud Task, you'd build and test standalone Task microservices in isolation. With the programming model, you can address use-cases ranging from ETL/ELT, data migration, or predictive-model-training etc.

Once you have an "n" no. of such applications, you could use Spring Cloud Data Flow's DSL to compose them into coherent Task/Batch pipelines. To use them in the DSL, you'd have to first register the coordinates of the standalone Task Apps.

Refer to the SCDF's Task overview and as well as the Task Developer Guide for more details - you could repeat it locally to follow along and learn the mechanics.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
  • Can a task put message to queue - say RabbitMQ? – mjs May 01 '18 at 22:31
  • You could use Spring Batch's [AmqpItemWriter](https://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/item/amqp/AmqpItemWriter.html) in your Task App to publish events to RabbitMQ. Also, if you are interested in Task lifecycle events, see [here](http://docs.spring.io/spring-cloud-dataflow/docs/1.4.0.RELEASE/reference/htmlsingle/#spring-cloud-dataflow-task-events). Likewise, you could launch a Task from [events](http://docs.spring.io/spring-cloud-dataflow/docs/1.4.0.RELEASE/reference/htmlsingle/#spring-cloud-dataflow-launch-tasks-from-stream), too. – Sabby Anandan May 01 '18 at 22:42
  • Thank you for the valuable information! Actually, I am still trying to deploy a sample spring batch job jar with main class having @EnableTask on to the local data flow server. I think post I can register/create task on it. Correct? – mjs May 03 '18 at 16:35
  • Yes, if the Task/Batch App runs locally (via `java -jar`), it should work in SCDF, too. You'd simply register the maven:// or file:// coordinates of the Jar file and create a Task definition with it in SCDF. I'd still recommend repeating the "Task Developer Guide" (see link in the post above) - if you make that work end to end, you can then switch to the custom Batch App. – Sabby Anandan May 03 '18 at 17:18
  • Thank you!! I could run a batch application as task on SCDF. Now I am trying to run https://spring.io/guides/gs/messaging-rabbitmq/ tutorial on SCDF but getting error ----> Failed to start bean 'taskLifecycleListener'; nested exception is java.lang.IllegalArgumentException: Invalid TaskExecution, ID 11 not found – mjs May 04 '18 at 18:31