NullPointerException When Launching Task in Spring Cloud Dataflow - in Kubernetes Environment

Question

I am setting up Spring Cloud Dataflow in Kubernetes Cluster environment. I faced this problem only on Kubernetes deployment. FYI, I am using server 2.7.0 (https://dataflow.spring.io/docs/2.7.0.SNAPSHOT/recipes/batch/batch-only-mode/) without Skipper. It works wonderfully when we deploy it locally and inside docker, locally. But, when we deploy it to Kubernetes, we can pull the example data or Docker image from private ACR, but when we try to launch it (even example data), we got this error :

java.lang.NullPointerException: null
    at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService.executeTask(DefaultTaskExecutionService.java:247)
    at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService$$FastClassBySpringCGLIB$$422cda43.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
    at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService$$EnhancerBySpringCGLIB$$af76387c.executeTask(<generated>)
    at org.springframework.cloud.dataflow.server.controller.TaskExecutionController.launch(TaskExecutionController.java:170)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:103)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

I tried using shell too from inside the pod - same problem, nullPointerException.

NullPointerException
org.springframework.cloud.dataflow.rest.client.DataFlowClientException: NullPointerException
at org.springframework.cloud.dataflow.rest.client.VndErrorResponseErrorHandler.handleError(VndErrorResponseErrorHandler.java:65)
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:782)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:740)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674)
at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:418)
at org.springframework.cloud.dataflow.rest.client.TaskTemplate.launch(TaskTemplate.java:168)
at org.springframework.cloud.dataflow.shell.command.TaskCommands.launch(TaskCommands.java:245)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
at java.lang.Thread.run(Thread.java:748)

Do you have any idea how to mitigate this? Thank you.

We fixed the problem - Apparently, the H2 database is not accessible when we deploy to kubernetes, and we need to set up external database, and set up privileges for creating new pods via Helm Chart installer. Now it works like charm for pulling docker image and running/scheduling task. — Fiorina Liberta, Sep 02 '20 at 01:36
Sorry you had trouble with it; while H2 is not a reliable DB for K8s deployment, we should handle this more gracefully on the backend. When you get a chance, please open an issue in the SCDF repo, we will look into ways to handle this exception gracefully. — Sabby Anandan, Sep 03 '20 at 01:12
And, congrats for eventually getting everything up and running. Kudos to you! — Sabby Anandan, Sep 03 '20 at 01:13
I am having exactly the same problem like you. Locally it works like a charm, in kubernetes we get a NPE. We are using external H2 Database in a Pod. Can you please explain why H2 database is not accessible when deployed to kubernetes? Our other applications running in other pods actually can connect to the H2 server, so i dont really get why spring-cloud cannot. — Manu, Apr 19 '21 at 07:49

NullPointerException When Launching Task in Spring Cloud Dataflow - in Kubernetes Environment

0 Answers0