[Feature]: Instrument dv2 observation cycle and scheduling with opentelemetry #8662
Description
Is there an existing issue for this?
- I have searched the existing issues
User Story
We have open telemetry instrumentation for all endpoints, i.e. REST request being send out and recieved. Additionally, we trace requests that go onto the RabbitMQs, code that is executed when a task is picked up from there is connected to the original request that has put the task onto the RMQ queue.
One thing that is missing is keeping track of code-calls that happen as a result of the observation cycle of the dv2 for dynamic-sidecars. For e.g. the creation of a dy-sidecar, all code is traced until a SchedulerData object is created which will be checked during the dv2 observation cycle.
This is - in my eyes - a prerequisite for the task discussed with @sanderegg and @bisgaard-itis to put the TraceContext into a docker service label of the dy-sidecar, so that the autoscaling microservice can pick this trace-context up. Before that can happen, as the dy-sidecar creation itself (calling docker service create) is happening inside the dv2 observation cycle, the trace has to be propagated into the observation cycle.
This issue keeps track of the ongoing discussions
Very early draft PR in #8661 .
Definition of Done
No response