Support per-zone PDB

[dimitarvdimitrov]

Background

The classic PodDisruptionBudget in kubernetes doesn't allow us to express rules like "restart as many pods as you need, as long as they belong to the same zone."

Problem

This means that we're stuck with PDB with maxUnavailable=1. For large deployments this means that kubernetes node recycling is extremely slow restarting one Mimir ingester at a time.

Proposal

Implement our own PDB version in the admission controller that the rollout-operator already is. If a pod should be disrupted, the rollout operator checks that there are no pods from different zones which are currently not available. If there are pods from the same zone, then the pod can also be disrupted.

What about partitions?

The classic PDB isn't ideal for partitions too. With partitions we can restart any partition replica as long as the other replica in a different zone is up.

[dimitarvdimitrov]

internal link (apologies): this is similar to how this was implemented in https://github.com/grafana/hosted-grafana/pull/5667

[charleskorn]

There was a bit of discussion of an idea like this in #163

[dimitarvdimitrov]

There was a bit of discussion of an idea like this in #163

A comment from Charles from that PR:

1. (my suggestion above) Add a validating webhook to rollout-operator that runs on eviction API requests to allow or block eviction requests for pods belonging to a StatefulSet managed by rollout-operator. The eviction API still does the deletion of the pods itself. This mirrors the behaviour of PDBs.
2. (this PR) Modify rollout-operator to respond to a label added to pods belonging to a StatefulSet managed by rollout-operator and delete them when it is safe to do so.

[deniszh]

If someone need immediate solution until this in not implemented yet: I solved similar issue running ZDB controller from aws/zone-aware-controllers-for-k8s . You can pick up my fork with golang and base image refreshed. Setup is quite straightforward but ping me if you have questions.
AWS blog about controllers - https://aws.amazon.com/blogs/opensource/speed-up-highly-available-deployments-on-kubernetes/