This is more of a design question rather than a technical issue.
I have two services and a MongoDB database. Both of the services are connected to the database.
Service A is the user-facing API that CRUDs the database state. Service B is the backend controller service that, anmong other things, orchestrates the work based on the database state.
The closest analogy I can think of is that service A is like kube-apiserver, and service B is like kube-scheduler.
If the user submits an order to the service A, it will be added to the database. This will generate an OTEL Span. Then, service B asynchronously receives the order (either through polling or a change stream) and starts the process to fulfil the order. This will also generate an OTEL Span.
Now, what is the best way to link or parent the spans together? Or at least make them a part of the same Trace. I can come up with two solutions but I don’t like any of them:
- Write the OTEL context into the database together with the order. Service B will then inherit that context. Simple, works, but pollutes the data.
- Rethink the communication between service A and service B to a more microservice-mesh style. Requires a redesign of everything.
Is there a common pattern to solving this problem? For example, is there a way to correlate the two Spans after they were created in OTEL?
