1

I am new to Uber Cadence framework and currently working on a project workflow management project using Cadence. I am seeing strange behavior all the workflows are having a timer for ~270hrs as part of the flow and I am not sure how that number is calculated and where that timer is coming from.

And the other issue is, once timer is fired, workflows are failing (not terminated) with UNHADLED_DECISION error. This exception is keep throwing and spamming the logs. Here is the stacktrace.

"com.uber.cadence.internal.replay.NonDeterminisicWorkflowError: Unknown DecisionId{decisionTarget=TIMER, decisionEventId=11}. The possible causes are a nondeterministic workflow definition code or an incompatible change in the workflow definition.\n\tat com.uber.cadence.internal.replay.DecisionsHelper.getDecision(DecisionsHelper.java:733)\n\tat com.uber.cadence.internal.replay.DecisionsHelper.handleTimerStarted(DecisionsHelper.java:451)\n\tat com.uber.cadence.internal.replay.ReplayDecider.processEvent(ReplayDecider.java:229)\n\tat com.uber.cadence.internal.replay.ReplayDecider.decideImpl(ReplayDecider.java:452)\n\tat com.uber.cadence.internal.replay.ReplayDecider.decide(ReplayDecider.java:385)\n\tat com.uber.cadence.internal.replay.ReplayDecisionTaskHandler.processDecision(ReplayDecisionTaskHandler.java:145)\n\tat com.uber.cadence.internal.replay.ReplayDecisionTaskHandler.handleDecisionTaskImpl(ReplayDecisionTaskHandler.java:125)\n\tat com.uber.cadence.internal.replay.ReplayDecisionTaskHandler.handleDecisionTask(ReplayDecisionTaskHandler.java:86)\n\tat com.uber.cadence.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:257)\n\tat com.uber.cadence.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:229)\n\tat com.uber.cadence.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:71)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"

enter image description here

Can somebody explain me what's happening here and what is this timer? Is there a way to handle this error/exception gracefully and avoid spamming the logs? There are thousand of workflows like this in the test environment and is there a way to terminate them all using Cadence Web or some other? Thanks in advance

Edit: I have two code blocks where I am using LocalDateTime and Workflow.sleep for waiting.

  1. If the workflow startTime is in the future, calculate the wait time and then sleep

    if (LocalDateTime.now(CampaignConstants.ZONE_ID_UTC).isBefore(myWorkflow.getStartDateTime())) { Duration waitDuration = Duration.between(LocalDateTime.now(), myWorkflow.getStartDateTime()); Workflow.sleep(waitDuration); }

  2. If the workflow step is to wait for specified time period, then call Workflow.sleep with the scarified time

    Integer waitPeriod = Integer.parseInt((String) props.get("waitPeriod")); ChronoUnit chronoUnit = ChronoUnit.valueOf((String) props.get("waitPeriodType")); Workflow.sleep(Duration.of(waitPeriod, chronoUnit));

Is this right way of implementing? Seems not the right way, so what's the proper way of implementing these functionalities. Thanks

Sharan Foga
  • 135
  • 8
siva
  • 53
  • 1
  • 10

2 Answers2

1

Not Cadence expert but since you tagged it also with "temporal-workflow" can give it a shot :)

The TimerStarted->Fired events seem to come from your workflow code. Check if you have workflow.Sleep in your code or create a timer and wait for it to complete in code.

After the timer fires you have a decision task that times out on ScheduleToStart timeout meaning the task was placed on task queue but was not picked up by one of your workers.

This task in then placed again on the global ("normal") task queue partition and most likely another one of your workers picked it up (check identity field on your WorkflowTaskStarted events). This worker did not have the execution history in its in-memory cache, meaning worker had to pull whole history from service and then perform internal workflow replay, which led to a non-deterministic error. Would check your code to see if you are using maybe system clock to calculate sleep durations or some other type on non-deterministic stuff in wf code. If you can share your code, could take a look. Hope this helps.

tsurdilovic
  • 61
  • 1
  • Just to add regarding "This exception is keep throwing and spamming the logs", the workflow task thats filing with non-deterministic error is being retried, and keeps failing thats why you see it in logs multiple times. – tsurdilovic Mar 04 '23 at 18:13
  • Thanks https://stackoverflow.com/users/1572920/tsurdilovic. I think what you pointed out is the exact issue. I am updating the question with the code block, please review. – siva Mar 05 '23 at 18:31
  • I assume that "props" is your workflow input, so using that to calculate your sleep duration is ok as workflow inputs are recorded in your execution event history. What is "myWorkflow.getStartDateTime()"? How is its return calculated? Don't use java local time in your code, use "Workflow. currentTimeMillis()" instead, as its deterministic. If getStartDateTime() method is deterministic too (share code please if you can) and with using Workflow. currentTimeMillis() instead of LocalDateTime would make duration calculation deterministic too imo. – tsurdilovic Mar 05 '23 at 20:26
  • Can you paste your personal chat link? May I can post the whole code and discuss privately. "myWorkflow" is a user defined flow chart in JSON and is passed input to the Workflow execution code. "myWorkflow.getStartDateTime()" is user specified startTime to start executing the flow and "props" is properties of a step in the user defined flow chart. – siva Mar 06 '23 at 06:16
  • Is there a way to terminate all hanging workflow (~10K workflows )? These are draining the resources – siva Mar 06 '23 at 06:58
  • maybe easiest is to do via cli, think for Cadence docs are: https://cadenceworkflow.io/docs/cli/#quick-start (batch start with terminate option and visibility query) – tsurdilovic Mar 06 '23 at 14:50
  • Using `Workflow.currentTimeMillis()` resolved the issue and the hanging workflows were terminated automatically after deploying the change. – siva Mar 15 '23 at 08:39
1

Your code is nondeterminstic as it uses LocalDateTime.now(CampaignConstants.ZONE_ID_UTC) instead of Workflow.currentTimeMillis. The workflow is by design doesn't fail, but is paused (by retrying workflow task which you see in the logs).

The first workflow task timeout (SCHEDULE_TO_START) is caused by disptaching the worklfow task on a sticky task queue (used for workflow caching). This issue is already fixed in Temporal.

Maxim Fateev
  • 6,458
  • 3
  • 20
  • 35