Pyrra for Saturation Metrics

[Navaneeth92]

Hi,

We are using Pyrra actively as the SLO solution for one of our project and we have Rabbit MQ involved in the critical flow. Two important saturation metrics need to be monitored from Rabbit MQ which are Queue Depth and Memory. These two indicators has been a cause for majority of P1 incidents. It would be good to have a format for handling saturation metrics [gauge format].

Thanks,
Navaneeth A

[metalmatze]

Thanks for starting this discussion. This is really interesting and exciting! So far I've mostly worked with SLOs in request-based but definitely want Pyrra to support Pipelines as well.
We should probably read through these examples SLIs for pipelines.

For me the hardest part about figuring support like this out is coming up with actual PromQL queries for existing metrics (or whether or not new metrics in systems need to be created).

Again, I'd love to explore the ideas and the more concrete your examples can be the more helpful it'll be for actually implementing something like that.

[Navaneeth92]

Thanks for the response , would be more than happy to elaborate.. please find below the screenshot for a quick overview of SLO's we are capturing.

Rabbit MQ Query for Memory using out of the box prometheus plugin looks like below:

rabbitmq_alarms_memory_used_watermark * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_idendity_info{rabbitmq_node="rabbit@nodename"}

This will result in 0 and 1 -> 0 means no alarm and 1 means alarm

alarm condition -> Rabbit MQ memory has breached the preset high watermark value.

this may seam like a unique query which only is applicable for Rabbit but we can build similar queries for other components as well, like

0 * (memory_metric{node="asd"} <= 80) or 1+(memory_metric{node="xyz"} >= 80). --> We can sue this if/else logic for any gauge metric to once again come down to 1s and 0s as result for easy slo calculation.

[paolostancato]

Hi,

I'm exploring pyrra for a similar use case: kafka lag (queue depth). I'm currently alerting on this metric:

sum(kafka_consumergroup_lag{consumergroup="kafka.consumer.group"}) / on() sum(rate(kafka_topic_partition_current_offset{topic="kafka.topic"}[1m]))

(messages in the queue) / (messages processed per minute)

that would give me the depth of the kafka queue in seconds, and it would be great for me to define SLOs of the form "queue depth below 60s" etc

[ArthurSens]

I wonder if we could offer some abstraction on top of BoolGauge SLO here.

Something around vector(1) and metric > threshold?

[paolostancato]

Hi Arthur,

thanks for your prompt response, it's very helpful! Actually I use sign and clamp to convert that metric to boolean (for other purposes) so I guess I could work around it. Will let you know if it works.

[ArthurSens]

If we find a query that works, it shouldn't be difficult to create a new type of SLO

[paolostancato]

Well, turns out I kinda hacked it: I created a recording rule like this:

clamp_min(
  sgn(
    <THRESHOLD> -
    (
      sum by (cluster, topic) (kafka_consumergroup_lag) /
      sum by (cluster, topic) (rate(kafka_topic_partition_current_offset[1m]))
    )
), 0)

and then a bool_gauge SLO :)

But it'd be great to be able to use it without the recording rule (and passing the threshold as an argument).