Image Render: Support Tracing

[lucychen-grafana]

Testing Instructons:

Local Setup:

1. Install tempo https://grafana.com/docs/tempo/latest/getting-started/docker-example/

• Run the dockerfile for tempo local storage in the local directory

2. Checkout this branch of grafana and run grafana-enterprise. Set otlp address in defaults.ini

[tracing.opentelemetry.otlp]
address = localhost:4317

4. Checkout this pr grafana-image-renderer branch. Run the server: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4318/v1/traces" node build/app.js server --port=8081 --config=dev.json to set env var
5. Open Localhost:3000; run a report or PDF creation, or any call to the image renderer service
6. Check traces: Explore -> Select ‘gdev-tempo’ datasource -> run query resource.service.name="image-renderer-service"
7. Check logs in the console to confirm trace_id and span_id is included in the logs

Deployment_tools pr for adding env vars

[CLAassistant]

All committers have signed the CLA.

[lucychen-grafana]

Confirm this is the case in the log output in terminal. You can check that trace_id, span_id, and trace_flags is included

[lucychen-grafana]

I followed OpenTelemetry Doc for Node.js to implement this

[lucychen-grafana]

the fs module is very noisy. It create 10+ traces with singular span so it did not make much sense to have it unless it's part of another trace.

There is a list of supported autoinstrumentation. We can opt out of any in the future we do not care to trace those libraries

[lucychen-grafana]

Create startTracing separately from the rest of initialization steps b/c we want to only capture traces if it's enabled

[lucychen-grafana]

OpenTelemetry needs to be initialized before the other modules (express, http, winston, etc) in order for it to auto capture telemetry trace data from those libraries so it must come first

[AgnesToulet]

This looks good! I tested it and this worked. However, I don't find the current traces very useful as there is a big blank gap in the middle.

An easy place to add some manual spans would be in the withTimingMetrics function. Something like that but adding some config checks:

  async withTimingMetrics<T>(step: string, callback: () => Promise<T>): Promise<T> {
    if (this.config.timingMetrics) {
      const endTimer = this.metrics.durationHistogram.startTimer({ step });
      const res = this.tracer.startActiveSpan(step, async (span) => {
        const cbRes = await callback();
        span.end();
        return cbRes;
      });
      endTimer();

      return res;
    } else {
      return callback();
    }
  }

The Browser class could initialize its tracer in its constructor (this should work for all browser implementations, e.g. with mode: "clustered"):

export class Browser {
  tracer: Tracer;

  constructor(protected config: RenderingConfig, protected log: Logger, protected metrics: Metrics) {
    this.log.debug('Browser initialized', 'config', this.config);
    this.tracer = trace.getTracer('browser');
  }
  ...

Also, I looked into what is missing to have a link between Grafana traces and image renderer traces and this simple change is working (to enable traces in Grafana, you need to set:

[tracing.opentelemetry.otlp]
# otlp destination (ex localhost:4317)
address = localhost:4317

). I will test it out a bit more and check if this need to be initialized only when tracing is enabled or if this is a no-op when tracing is disabled.

[AgnesToulet]

This whole file should be formatted using Prettier linter (we don't have a check for that in CI but we should do our best).

[lucychen-grafana]

[tracing.opentelemetry.otlp]

otlp destination (ex localhost:4317)

address = localhost:4317

@AgnesToulet setting this address is enable traces in Grafana in dev env right? Traces should already be enabled in Cloud via Faro

[AgnesToulet]

Yes it's for your local environment. It's enabled by default on Cloud (Faro is for frontend monitoring, its configuration is in a different section of the ini file, I don't think both are related).