Gateway timeout for prometheus, grafana, alertmanager

[Jeter-work]

Setting up my first kubernetes cluster. k3s on Raspberry Pis running Rocky.

I followed the instructions on the readme, in the k3s section. First manually, loading the manifests 'manually'. Then I used make teardown and redid it with make deploy.

kubectl get ingress -n monitoring shows all three with ..nip.io

The first problem was that my router refused to resolve nip.io. So I changed the DNS on my windows pc to cloudflare public dns. SO it resolved to the local IP and started trying to load.

I've tried them with http, https, without any prefix. I tried them with the ip address directly, so .. I tried in edge. I even tried chromium in the ubuntu vm I use for ansible. All of the windows computer get gateway timeout. The vm gets dns resolution issue. Finally I tried my phone. Same as the VM. No dns.

I understand teh no dns ones. But when I use https://prometheus.192.168.x.y it should freaking go without DNS

It's always DNS right? But I'm sure I did something wrong too.

[Jeter-work]

# cat vars.jsonnet 
{
  _config+:: {
    namespace: 'monitoring',
  },
  // Enable or disable additional modules
  modules: [
    {
      // After deployment, run the create_gmail_auth.sh script from scripts dir.
      name: 'smtpRelay',
      enabled: false,
      file: import 'modules/smtp_relay.jsonnet',
    },
    {
      name: 'armExporter',
      enabled: true,
      file: import 'modules/arm_exporter.jsonnet',
    },
    {
      name: 'upsExporter',
      enabled: false,
      file: import 'modules/ups_exporter.jsonnet',
    },
    {
      name: 'metallbExporter',
      enabled: false,
      file: import 'modules/metallb.jsonnet',
    },
    {
      name: 'nginxExporter',
      enabled: false,
      file: import 'modules/nginx.jsonnet',
    },
    {
      name: 'traefikExporter',
      enabled: true,
      file: import 'modules/traefik.jsonnet',
    },
    {
      name: 'elasticExporter',
      enabled: false,
      file: import 'modules/elasticsearch_exporter.jsonnet',
    },
    {
      name: 'speedtestExporter',
      enabled: false,
      file: import 'modules/speedtest_exporter.jsonnet',
    },
  ],

  k3s: {
    enabled: false,
    master_ip: ['192.168.86.230'],
  },

  // Domain suffix for the ingresses
  suffixDomain: '192.168.86.29.nip.io',
  // Additional domain suffixes for the ingresses.
  // For example suffixDomain could be an external one and this a local domain.
  additionalDomains: [],
  // If TLSingress is true, a self-signed HTTPS ingress with redirect will be created
  TLSingress: true,
  // If UseProvidedCerts is true, provided files will be used on created HTTPS ingresses.
  // Use a wildcard certificate for the domain like ex. "*.192.168.99.100.nip.io"
  UseProvidedCerts: false,
  TLSCertificate: importstr 'server.crt',
  TLSKey: importstr 'server.key',

  // Persistent volume configuration
  enablePersistence: {
    // Setting these to false, defaults to emptyDirs.
    prometheus: false,
    grafana: false,
    // If using a pre-created PV, fill in the names below. If blank, they will use the default StorageClass
    prometheusPV: '',
    grafanaPV: '',
    // If required to use a specific storageClass, keep the PV names above blank and fill the storageClass name below.
    storageClass: '',
    // Define the PV sizes below
    prometheusSizePV: '2Gi',
    grafanaSizePV: '20Gi',
  },

  // Configuration for Prometheus deployment
  prometheus: {
    retention: '15d',
    scrapeInterval: '30s',
    scrapeTimeout: '30s',
  },
  grafana: {
    // Grafana "from" email
    from_address: '[email protected]',
    // Plugins to be installed at runtime.
    //Ex. plugins: ['grafana-piechart-panel', 'grafana-clock-panel'],
    plugins: [],
    //Ex. env: [ { name: 'http_proxy', value: 'host:8080' } ]
    env: []
  },
}

# kubectl get pods -n monitoring
NAME                                  READY   STATUS    RESTARTS      AGE
prometheus-operator-67755f959-z6t66   2/2     Running   0             22m
arm-exporter-xsk5t                    2/2     Running   0             21m
arm-exporter-sj7wv                    2/2     Running   0             21m
arm-exporter-f2jzx                    2/2     Running   0             21m
arm-exporter-k54qw                    2/2     Running   0             21m
arm-exporter-f8rsg                    2/2     Running   0             21m
arm-exporter-7s2pg                    2/2     Running   0             21m
arm-exporter-fm7nw                    2/2     Running   0             21m
arm-exporter-wqhbw                    2/2     Running   0             21m
arm-exporter-dmnkm                    2/2     Running   0             21m
arm-exporter-8bx8l                    2/2     Running   0             21m
kube-state-metrics-6cb6df5d4-ft22r    3/3     Running   0             21m
node-exporter-qrrd4                   2/2     Running   0             21m
node-exporter-smspp                   2/2     Running   0             21m
node-exporter-c5hhr                   2/2     Running   0             21m
node-exporter-m5dfg                   2/2     Running   0             21m
node-exporter-25tvh                   2/2     Running   0             21m
node-exporter-xnpvq                   2/2     Running   0             21m
node-exporter-9m4p9                   2/2     Running   0             21m
node-exporter-pdvnm                   2/2     Running   0             21m
alertmanager-main-0                   2/2     Running   0             21m
node-exporter-ffdx2                   2/2     Running   0             21m
node-exporter-lfh7r                   2/2     Running   0             21m
prometheus-adapter-585b57857b-7jcp8   1/1     Running   0             21m
grafana-7bc4784744-4r8tx              1/1     Running   0             21m
prometheus-k8s-0                      3/3     Running   1 (21m ago)   21m

kubectl get ingress -n monitoring
NAME                CLASS    HOSTS                                ADDRESS                                                                                                                                                 PORTS     AGE
alertmanager-main   <none>   alertmanager.192.168.86.230.nip.io   192.168.86.230,192.168.86.231,192.168.86.232,192.168.86.233,192.168.86.234,192.168.86.235,192.168.86.236,192.168.86.237,192.168.86.238,192.168.86.239   80, 443   67s
grafana             <none>   grafana.192.168.86.230.nip.io        192.168.86.230,192.168.86.231,192.168.86.232,192.168.86.233,192.168.86.234,192.168.86.235,192.168.86.236,192.168.86.237,192.168.86.238,192.168.86.239   80, 443   67s
prometheus-k8s      <none>   prometheus.192.168.86.230.nip.io     192.168.86.230,192.168.86.231,192.168.86.232,192.168.86.233,192.168.86.234,192.168.86.235,192.168.86.236,192.168.86.237,192.168.86.238,192.168.86.239   80, 443   67s

[Jeter-work]

Can I set the ingress to 192.168.86.230/prometheus ?

[gregbrant2]

Similar to @Jeter-work I've deployed cluster-monitoring to three Pi 2Bs and I have issues. I'm posting here because my 404 issue smells similar, and I've got a feeling the 404 is a result of the other symptoms I'm seeing. Any suggestions greatly appreciated.

404s

When I try and hit any of the ingress URLs I get

404 page not found

I note that when I get a response from kubectl get ingress -n monitoring the ADDRESS column is empty:

$ kubectl get ingress -n monitoring
NAME                CLASS    HOSTS                               ADDRESS   PORTS     AGE
alertmanager-main   <none>   alertmanager.192.168.1.142.nip.io             80, 443   27m
grafana             <none>   grafana.192.168.1.142.nip.io                  80, 443   27m
prometheus-k8s      <none>   prometheus.192.168.1.142.nip.io               80, 443   27m

Locked up

When the deployment eventually finishes I can't get to any of the services. Initially I can get responses from various kubectl calls e.g.

$ kubectl get ingress -n monitoring
NAME                CLASS    HOSTS                               ADDRESS   PORTS     AGE
alertmanager-main   <none>   alertmanager.192.168.1.142.nip.io             80, 443   27m
grafana             <none>   grafana.192.168.1.142.nip.io                  80, 443   27m
prometheus-k8s      <none>   prometheus.192.168.1.142.nip.io               80, 443   27m

$ kubectl get services -A
NAMESPACE     NAME                                           TYPE           CLUSTER-IP      EXTERNAL-IP                                 PORT(S)                        AGE
default       kubernetes                                     ClusterIP      10.43.0.1       <none>                                      443/TCP                        9h
kube-system   kubelet                                        ClusterIP      None            <none>                                      10250/TCP,10255/TCP,4194/TCP   8h
kube-system   kube-dns                                       ClusterIP      10.43.0.10      <none>                                      53/UDP,53/TCP,9153/TCP         9h
kube-system   metrics-server                                 ClusterIP      10.43.208.2     <none>                                      443/TCP                        9h
kube-system   traefik-prometheus                             ClusterIP      10.43.94.150    <none>                                      9100/TCP                       9h
monitoring    prometheus-operator                            ClusterIP      None            <none>                                      8443/TCP                       30m
monitoring    alertmanager-main                              ClusterIP      10.43.12.137    <none>                                      9093/TCP                       30m
monitoring    alertmanager-operated                          ClusterIP      None            <none>                                      9093/TCP,9094/TCP,9094/UDP     30m
monitoring    arm-exporter                                   ClusterIP      None            <none>                                      9243/TCP                       30m
monitoring    grafana                                        ClusterIP      10.43.46.244    <none>                                      3000/TCP                       28m
monitoring    kube-state-metrics                             ClusterIP      None            <none>                                      8443/TCP,9443/TCP              26m
monitoring    node-exporter                                  ClusterIP      None            <none>                                      9100/TCP                       25m
monitoring    prometheus-adapter                             ClusterIP      10.43.215.116   <none>                                      443/TCP                        23m
kube-system   kube-controller-manager-prometheus-discovery   ClusterIP      None            <none>                                      10252/TCP                      22m
kube-system   kube-dns-prometheus-discovery                  ClusterIP      None            <none>                                      9153/TCP                       22m
kube-system   kube-scheduler-prometheus-discovery            ClusterIP      None            <none>                                      10251/TCP                      22m
kube-system   traefik                                        LoadBalancer   10.43.212.164   192.168.1.141,192.168.1.142,192.168.1.143   80:32624/TCP,443:30738/TCP     9h
monitoring    prometheus-operated                            ClusterIP      None            <none>                                      9090/TCP                       21m
monitoring    prometheus-k8s                                 ClusterIP      10.43.131.185   <none>                                      9090/TCP                       21m
monitoring    speedtest-exporter                             ClusterIP      10.43.210.140   <none>                                      9798/TCP                       21m

$ kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS      RESTARTS        AGE
kube-system   helm-install-traefik--1-4rnfs            0/1     Completed   0               6h39m
kube-system   svclb-traefik-f9jvz                      2/2     Running     0               6h34m
kube-system   metrics-server-9cf544f65-9t87w           1/1     Running     0               6h39m
kube-system   traefik-7b8b884c8-z45w2                  1/1     Running     1 (6h44m ago)   9h
monitoring    node-exporter-x25jt                      2/2     Running     0               26m
monitoring    node-exporter-dbmb6                      2/2     Running     0               26m
monitoring    arm-exporter-kdz4l                       2/2     Running     0               30m
monitoring    alertmanager-main-0                      2/2     Running     0               31m
monitoring    arm-exporter-jfvh7                       2/2     Running     0               30m
monitoring    kube-state-metrics-6cb6df5d4-m7hpq       3/3     Running     0               26m
kube-system   svclb-traefik-82kw5                      2/2     Running     0               6h34m
kube-system   local-path-provisioner-64ffb68fd-2vcxh   1/1     Running     0               6h39m
kube-system   coredns-85cb69466-j5jw7                  1/1     Running     0               6h39m
monitoring    prometheus-adapter-585b57857b-4p25d      1/1     Running     0               23m
monitoring    speedtest-exporter-7999d7b765-dzmwr      1/1     Running     0               22m
monitoring    node-exporter-whklt                      2/2     Running     2 (5m41s ago)   26m
kube-system   svclb-traefik-b86p2                      2/2     Running     2 (5m41s ago)   6h34m
monitoring    prometheus-k8s-0                         3/3     Running     4 (5m41s ago)   21m
monitoring    arm-exporter-f2fbs                       2/2     Running     2 (5m41s ago)   30m
monitoring    grafana-55998c6bfc-s5t8v                 1/1     Running     1               29m
monitoring    prometheus-operator-67755f959-5jt5k      2/2     Running     4 (2m59s ago)   31m

but after a little while the CPU like on the master comes on solid and I start to get an error from kubectl like

$ kubectl get pods -A


Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)

or

$ kubectl get ingress -n monitoring
Unable to connect to the server: net/http: TLS handshake timeout

If I ssh onto the master and run top I can see various processes (k3s-server, grafana-server, prometheus, containerd, etc) using between 10% and ~90% CPU (of 400% as 4 cores, right?).

The CPU doesn't look pegged in top but it feels it. ssh is super sluggish and that CPU activity led is on solid.

When I reboot the master node I see

[FAILED] Failed to start Leightweight Kubernetes.
See `systemctl status k3s.service` for details.

then if I do sudo systemctl start k3s.service, after this is where the CPU led comes on solid and the system becomes super sluggish.

Does anyone have any ideas?