Measure time spent in different parts of application

[ngoral]

Hi!

I'd like to build a graph showing how many time of request is spent in different parts of app, e.g. Redis, Postgres, external requests with Net::HTTP and so on. I am using Prometheus and Grafana for graphs.
But now I can't see an easy and elegant way to do it, other that try to parse span names specific for each service and then group by them, but this solution does not look very stable.
I thought of passing otel.library.name into Prom, and them grouping by it, since it is very clearly defining a part of an app we are in, but it seems it can't be passed for some reason (may be because of being some kind of system attribute).

Is there a more reliable and elegant way? Or am I trying to do something otel is not suitable for?

[arielvalentin]

The Ruby SDK does not currently have a metrics implementation so you'll need some help from the OTel Collector to generate metrics from span data. Here's a tutorial on how to set up the OTel collector to export metrics to prometheus:

https://grafana.com/blog/2022/05/10/how-to-collect-prometheus-metrics-with-the-opentelemetry-collector-and-grafana/

Here's a tutorial that describes how to configure the SpanMetricsProcessor, which generates metrics from span data:

https://automationrhapsody.com/distributed-system-observability-extract-and-visualize-metrics-from-opentelemetry-spans/

Hope that helps.

[ngoral]

Thank you! However, my question was not on Prom, but on how to get info I need from spans that I have. Or how to add this info to span. Now I don't know a good way of saying to which part of my app (Rails, external request, db, ...) each span belongs. otel.library.name seems to be a good source of this information, but I can't pass it to Prom in any way. It seems to me, it's detected as some special key that can't be passed (may be because it starts with otel).
So I wonder if someone tried to do the same, or just knows how to add this info to span without having to patch gem, so it copies otel.library.name to another attribute. May be I'm missing something and there is other way different from looking at otel.library.name

[arielvalentin]

I am having difficulty helping because I do not understand your question.

However, my question was not on Prom, but on how to get info I need from spans that I have. Or how to add this info to span.

AFAIK Prometheus uses a TSDB, which requires a specific format that is closer to metrics than traces. If you want to visualize span data in Grafana you will need a tracing engine and datastore like Jaeger https://www.jaegertracing.io/

You may store metric data in Prometheus that you can derive from trace data but you have to convert it. See the links above on how to do that.

Now I don't know a good way of saying to which part of my app (Rails, external request, db, ...) each span belongs.otel.library.name seems to be a good source of this information, but I can't pass it to Prom in any way. It seems to me, it's detected as some special key that can't be passed (may be because it starts with otel).

There are existing attributes on spans like name or semantic conventions like db.system and there are some reserved key prefixes for external systems like otel. but you would have to consult the product documentation on that. https://opentelemetry.io/docs/reference/specification/common/attribute-naming/#otel-namespace

So I wonder if someone tried to do the same, or just knows how to add this info to span without having to patch gem, so it copies otel.library.name to another attribute. May be I'm missing something and there is other way different from looking at otel.library.name

Here are a few options:

1. You may write a custom processor with an on_start method to transform the spans but the SDK is not designed with those kinds of transformations in mind.
2. You may also write a custom exporter that amends data when it is exporting it to your backend.
3. Deploy an otel collector and use the built in attribute processor to copy the attributes to a new field: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor