0

We have registered the activities with auto-heartbeat configuration EnableAutoHeartBeat: true and also configured the activity option config HeartbeatTimeout: 15Min in the activity implementation.

  1. Do we still need to explicitly send heart-beat using activity.heartbeat() or is it automatically taken care by the go-client library?
  2. If its automatic, then what will happen if the Activity is waiting for external API response say >15Min delay?
  3. What will happen during the activity heart-beat if the worker executing the activity crashes or killed?
  4. Will Cadence retry the activities due to heart-beat failures?
Suneeth Kumar
  • 1

1 Answers1

1

  1. No, the SDK will take care of it with this config.

  2. The auto heartbeat will send heartbeat for every interval — the interval is 80% * Heartbeat timeout(15 minutes in your case) so that the activity won’t get timeout as long as the activity worker is still live.

So you should use a smaller heatbeat timeout, ideally 10~20s is the best.

  1. The activity will fail with “heartbeat timeout “

  2. Yes if you have set a retry policy .

See my other answer for retry policy

How to set proper timeout values for Cadence activities(local and regular activities, with or without retry)?

Example

Let say your activity implementation is waiting on AWS SDK API for 2 hours (max API timeout configured) --

You should still use 10-20 s for heartbeat timeout, and also use 2 hours for activity start to close timeout. Heartbeat timeout is for detecting the host is not live anymore so that the activity can be restarted as early as possible. Imagine this case: Because the API takes 2 hours, the activity worker got restarted during the 2 hours. If the HB timeout is 15 minutes, then Cadence will retry this activity after 15 minutes. If HB timeout is 10s, then Cadence can retry it after 10s, because it will get HB timeout within 10 seconds.

Long Quanzheng
  • 2,076
  • 1
  • 10
  • 22