Pluggable Router Implementations

[danehans]

I currently use a Gateway API implementation with an inference extension to perform similar functionality as the vLLM router. I would like to use vllm-stack but with my current router implementation. Will you consider integrating with inference extension as a supported router implementation?

[KuntaiDu]

Oh good suggestion. I am less familiar with k8s inference extension, so I'll involve @ApostaC and we will investigate and get back to you.

[ApostaC]

Thanks for asking!
We don't have an immediate plan to integrate with the inference extension API for now. But will absolutely consider it as this project grows.

An alternative solution is to disable the router and directly connect the gateway API to the vLLM service. (Note that currently the helm chart does not support disabling router via values.yaml, so maybe create an issue for this if you want so that we can work on that?)

[danehans]

@ApostaC thanks for the feedback. I created #66 to support disabling the router via Helm values. I would like to keep this issue open so production-stack can consider supporting Gateway API with inference extension.