0

I know there are libraries for tracing requests in distributed systems based on OpenTracing and OpenTelemetry; these all work because the requests are connected/chained(microservices talking to each other). How to trace when systems/services are disconnected?

Service X calls Service Y. Y performs some processing then writes to a shared location . A component Z polls shared location(after hrs) and does processing.

I want to know if for each of the request generated by X, the status at each of these 3 systems.

A simple way to track this would be have a table and update DB Entries. However considering the fact that I might need to retry certain things, I had thought of introducing a workflow engine(Cadence) that opens up a workflow at Service X. Then at each of the Services the workflow is updated. If Request Reaches Y, a step in w/f is completed. If it does not reach Z the workflow is left open and after some time I could generate an Email etc. saying the workflow failed.

I need help in understanding if this is the right way and/or different methods to do this.

Magellan
  • 71
  • 1
  • 2
  • 11

1 Answers1

1

This is one of the most typical use cases of Cadence Workflow. The recommended solution is let service X to start a workflow. It will execute an activity to make a service call to service Y. If possible, let Component Z to send a signal to the workflow, telling that the processing is done. Or you can have another activity to keep polling the status.

Note that back off retry can be done easily for activities supported by Cadence.

Then use a timer in the workflow to trigger an activity to send email.

Finally the status of this workflow is still as a local variable of the workflow. You can implement a Query handler in the workflow to get status. Or you can implement a background activity to report the progress proactively .

Open tracing library framework is for solving another problem. It is to measuring and analyzing service dependency and latency. In fact, Cadence workflow also supports open tracing here is the example. It is for service level, not for a particular requests or processing.

Long Quanzheng
  • 2,076
  • 1
  • 10
  • 22
  • I forgot to mention explicitly but I need to get an overall latency as well; like for example when the Request is initiated at X how long it takes to Reach to Z. Thanks. – Magellan Mar 10 '21 at 04:55
  • There are a couple of ways doing it. Within a workflow You can get the current timestamp of any step using Workflow.Now() API. If Z is a separate component that need to send a signal back, let it send a object with a time stamp in it. So that you can use time stamp to calculate how long it takes to reach Z. Or In Z query the start time and do the calculations. If Z is also an activity of the workflow, you can also use the open tracing framework – Long Quanzheng Mar 10 '21 at 17:38
  • Cadence also emit a lot of standard metric like workflow and activity latency. You can get them for free – Long Quanzheng Mar 10 '21 at 17:51