2

We are repeatedly seeing resource quota limitation issues in logs and Task jobs fail on the SCDF running on Kubernetes. Problem is, there are so many pods in "running" status even after they completed. I understand, SCDF does not delete the pods and it is developer's responsibility to cleanup.

Even when I run the Task Execution Cleanup from SCDF dashboard UI, it only cleans up the execution logs and task form UI but the pods created by that task still remain. Is this expected ? Shouldn't Task Execution Cleanup also delete the pods ? We are using Spring-Cloud-Dataflow-Server 2.4.2 Release.

Is there a way to cleanup the pods right after the execution is complete ? Any best practices here ?

Sudhakar Master
  • 43
  • 7

1 Answers1

4

Method - 1

You can clean up task executions by using restful api provided by spring-cloud-dataflow.

Request Structure

DELETE /tasks/executions/{ids}?action=CLEANUP,REMOVE_DATA HTTP/1.1
Host: localhost:9393

Fire DELETE request.

http://<stream-url>/tasks/executions/<task-ids-seperated-by-coma>?action=CLEANUP,REMOVE_DATA

eg: http://localhost:9393/tasks/executions/1,2?action=CLEANUP,REMOVE_DATA

Delete Task Execution

Note: Above api will clean up resources that were used to deploy tasks and delete the data associated with task executions from the underlying persistence store.

CLEANUP : clean up the resources

REMOVE_DATA : remove data from persistence store.

You can either pass both actions or single action depends on your use-case.

Method - 2

Using spring cloud dataflow shell.

Enter into the spring-cloud-dataflow shell and execute below command.

task execution cleanup --id <task-id>

eg: task execution cleanup --id 1

Cleanup task execution from spring-cloud-dataflow shell

Other option (Applicable for kubernetes platform only)

If you wan't to delete all completed pods then you can delete using kubectl tool.

kubectl delete pod --field-selector=status.phase==Succeeded -l role=spring-app -n <namespace-where-tasks-launched>

If you wan't to delete all pods with Error status then execute below command

kubectl delete pod --field-selector=status.phase==Failed -l role=spring-app -n <namespace-where-tasks-launched>
pvrforpranavvr
  • 2,708
  • 2
  • 24
  • 34