Render: Support tracing

[andsens]

Grafana supports both reporting spans and forwarding tracing IDs. It would be great if the image renderer could do so as well (or at least forward the tracing ID back to grafana).
That would enable an unbroken trace from an image rendering request (or even before that) all the way to the datasource like e.g. prometheus, which has recently added support for tracing as well.
Right now when rendering images, you see two independent traces being reported: One for the original request, and one by the image-renderer when it calls back to grafana.

[lucychen-grafana]

Info:
POC from Eze #562
Doc on setting up traces https://grafana.com/docs/grafana-cloud/send-data/traces/set-up/

[lucychen-grafana]

Reading material recommended:

• Overview of traces https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/traces/

Local Setup:

1. Install tempo https://grafana.com/docs/tempo/latest/getting-started/docker-example/
2. Run the dockerfile for tempo local storage
3. Can use grafana docker image in the dockerfile yaml file OR run local grafana repo
4. Open Localhost:3000; run a report or PDF creation, any that makes call to the image renderer service
5. Check traces - Explore -> Select ‘gdev-tempo’ datasource -> run query resource.service.name="image-renderer-service"

[lucychen-grafana]

Frontend Implementation:
Example: [Faro](https://grafana.com/docs/grafana-cloud/monitor-applications/frontend-observability/instrument/tracing-
instrumentation

Follow OpenTelemetry Doc for Node.js

• 1. Add OpenTelemetry (OTEL) Dependencies
• 2. Create a tracing configuration script (tracing.js) to set up the OpenTelemetry SDK
• 3. Configure ENV var for OTEL endpoint; add new config to deployment_tools
• 4. Find out how to test in dev/ops before prod
• 5. Docs

Grafana:
Starts the trace: Trace ID = 123456789abcdef123456789abcdef12.
Sends the request to the image renderer with the trace headers.

Image Renderer:
Receives the trace ID in the headers: 123456789abcdef123456789abcdef12.
Reports its spans with the same trace ID.
Sends the callback request back to Grafana with the same trace ID.

Grafana (Callback):
Links the callback operation to the same trace ID: 123456789abcdef123456789abcdef12.

Sample:
https://github.com/grafana/grafana-com/blob/8de5622df51e54f049e6ed97454361a7a7f4c9c1/packages/grafana-com-lib-api/src/o11y/tracing.ts

https://github.com/grafana/intro-to-mltp/blob/149951ea1b8cd5aa997eabc444b8ed8fcf5191fe/source/common/tracing.js

https://github.com/grafana/deployment_tools/blob/42b23ee8495a3a31f2cbc0591c8f6bac1b7d60c5/ksonnet/lib/smtprelay/smtprelay.libsonnet#L27-L31