PV values too small and/or "inverted"?

[badnetmask]

Hello all,

First of all, let me thank the developer and the maintainers for this great project.

Today I faced an issue where the Prometheus PV filled out and stopped recording metrics. Looking ar vars.jsonnet, I have a feeling like the PV sizes are too small and/or "inverted":

cluster-monitoring/vars.jsonnet

Lines 76 to 77 in 9b70233

	prometheusSizePV: '2Gi',
	grafanaSizePV: '20Gi',

With merely 2 days of operation on a 2 node cluster, the 2Gi Prometheus PV filled completely, while the 20Gi Grafana is only using 450Mi.

In other people's experiences, what would be good values for the PVs here?

[carlosedp]

The default values in the config are samples or placeholders for small test clusters. This highly depends on the amount of nodes and data generated by your applications. I'd go with something like 10-20Gb for Prometheus. The Grafana PV can be reduced to 2-4Gb and might never get filled.

[badnetmask]

Thanks, I did "invert" by myself, but I did not have enough time go do dig deep into the configs. Is there some sort of rotation/retention time in the Prometheus configs?

[carlosedp]

by default Prometheus retains data for 15 days. You can config this in vars.jsonnet in the retention parameter.