[synthetic-monitoring-agent] fix deployment not starting on update/auto-scaling

[Iridias]

Retry #2994

The implementation of the agent does not seem to allow for concurrency.
Thus scaling the deployment - either on helm-upgrade or auto-scaling - will result in the new PODs to never become ready!

So any helm-upgrade will run into a timeout and then abort.

In the logs you'll find the following messages (if debug is enabled):

{"level":"info","program":"synthetic-monitoring-agent","subsystem":"updater","error":"registering probe with synthetic-monitoring-api, response: probe already exists","was_connected":false,"connection_state":"READY","time":1708611775289,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/checks/checks.go:259","message":"broke out of loop"}
{"level":"warn","program":"synthetic-monitoring-agent","subsystem":"updater","error":"registering probe with synthetic-monitoring-api, response: probe already exists","connection_state":"READY","time":1708611775289,"caller":"github.com/grafana/synthetic-monitoring-agent/internal/checks/checks.go:309","message":"handling check changes"}

With emphasis on: response: probe already exists

To fix that, I changed the Deployment to a StatefulSet, as k8s ensures, that the old POD is killed/deleted before spawning the new one.
I also removed all the autoscaling-resources, as they're not useful anyway.

And of course, I also successfully tested the changes on one of our clusters.

[Sheikh-Abubaker]

@Iridias Could you please resolve the Conflicts

[CLAassistant]

All committers have signed the CLA.

[Sheikh-Abubaker]

@zanhsieh what do you think about this ? I mean there are other charts as well which are using hpa, how are they resolving this ?

[Iridias]

there are other charts as well which are using hpa, how are they resolving this ?

I'd assume, they don't have to solve this - and that the apps, they're for, are HA-capable.
The synthetic-monitoring-agent is not, and does not allow for more than one client/pod per probe.

That's still true with the current version 0.31.0