Monitor Services using annotations

[julianfbeck]

Hey Carlos, hey community,

thank you for creating this awesome repo, I love that it worked first try.
Coming from other Prometheus setups, I know that I can scrape services using annotations like:

annotations:
   prometheus.io/scrape: 'true'
   prometheus.io/path: /metrics
   prometheus.io/port: '8081'

This doesn't seem to work on this project, so checked the scrape_rules on the Prometheus dashboard and saw that they were missing. I looked around in the generated manifest and if i understood it correctly, you used the kind: ServiceMonitors to tell Prometheus to monitor different applications.

How can i add the custom scrape rules to the Prometheus config. I tried using a config-map and point the prometheus-operator to it using : - --config.file=/etc/config/prometheus.yml in the prometheus-operator-deployment.yaml deployment, but this generated an error, that the --config.file argument is not valid. Somehow, this is disabled in your custom image.

Or should I just use the ServiceMonitor CRDs to define what resources to monitor, or what are the advantages/disadvantages to using annotations.
Thanks, Julian

EDIT: If im correct, i can add this functionality the following way?:
https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md

[julianfbeck]

Never mind, I figured it out using the additional crape config (y) and adding:

- job_name: kubernetes-service-endpoints
  kubernetes_sd_configs:
    - role: endpoints
  relabel_configs:
    - action: keep
      regex: true
      source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
    - action: replace
      regex: (https?)
      source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
      target_label: __scheme__
    - action: replace
      regex: (.+)
      source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
      target_label: __metrics_path__
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - action: replace
      source_labels:
        - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
    - action: replace
      source_labels:
        - __meta_kubernetes_service_name
      target_label: kubernetes_name
    - action: replace
      source_labels:
        - __meta_kubernetes_pod_node_name
      target_label: kubernetes_node
- job_name: kubernetes-service-endpoints-slow
  kubernetes_sd_configs:
    - role: endpoints
  relabel_configs:
    - action: keep
      regex: true
      source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
    - action: replace
      regex: (https?)
      source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
      target_label: __scheme__
    - action: replace
      regex: (.+)
      source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
      target_label: __metrics_path__
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - action: replace
      source_labels:
        - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
    - action: replace
      source_labels:
        - __meta_kubernetes_service_name
      target_label: kubernetes_name
    - action: replace
      source_labels:
        - __meta_kubernetes_pod_node_name
      target_label: kubernetes_node
  scrape_interval: 5m
  scrape_timeout: 30s
- job_name: kubernetes-pods
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - action: keep
      regex: true
      source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
    - action: replace
      regex: (.+)
      source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
      target_label: __metrics_path__
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      source_labels:
        - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
    - action: replace
      source_labels:
        - __meta_kubernetes_pod_name
      target_label: kubernetes_pod_name
- job_name: kubernetes-pods-slow
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - action: keep
      regex: true
      source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
    - action: replace
      regex: (.+)
      source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
      target_label: __metrics_path__
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      source_labels:
        - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
    - action: replace
      source_labels:
        - __meta_kubernetes_pod_name
      target_label: kubernetes_pod_name
  scrape_interval: 5m
  scrape_timeout: 30s

[Robert-Zacchigna]

@julianfbeck Can you clarify how you did this? Im running into the same issue but im unable to get the additional scraping metrics to show up.

[McMac]

@julianfbeck, how did you load this config? (I don't have access to the prometheus.yml)

All I need is to basically add 3 lines

  - job_name: 'fritzbox_exporter'
    static_configs:
      - targets: ['192.168.1.100:9042']

(shown in the instructions here: Fritzbox Monitoring mit Grafana Dashboard)