Making the producer stalling configurable

[jing-flowdesk]

Proposed change

Making the producer stalling configurable.

Currently, when you have a slow consumer on a subject, it can make the server throttle the provider.
ex:

// If we are a client and we detect that the consumer we are
// sending to is in a stalled state, go ahead and wait here
// with a limit.
if c.kind == CLIENT && client.out.stc != nil {
client.stalledWait(c)
}

I read on another issue that it was to protect GW and Routes ?
The proposal would be to make this configurable by allowing people to switch this off if they need.

Use case

Better handling event spikes by not stalling the producers if there are a slow consumer on some topic.

Contribution

The change looks not huge here but we would love to have some more inputs before saying we can do it.

[neilalexander]

The stall gate here is to prevent the server from spiking in memory usage in queues when the subscriber is failing to keep up, otherwise the server could potentially OOM and lose it all anyway.

If your usage pattern features very spiky producers but stable or throttled subscribers, you may want to look at funnelling the data through streams instead.

[jing-flowdesk]

Thank you @neilalexander ,

are you talking about https://docs.nats.io/nats-concepts/jetstream/streams ?
Jetstream unfortunately is not matching our needs because notably of the increased latency when activating it.

We were wondering if we could have "Fire and Forget" operating mode where the producer is publishing at its own speed and never getting throttled. The consumer may or may not receive the message depending of its healthness.