Public Preview: Default Inference with vLLM with AI Toolchain Operator Add-on

By default, the AI toolchain operator add-on now deploys model inferencing workloads using the high-throughput vLLM serving engine. vLLM offers a significant speed-up to process incoming requests and flexibility to use OpenAI-compatible APIs, DeepSeek R1 models, and a range of pre-trained HuggingFace models. Alternatively, you can switch from vLLM to HuggingFace Transformers for KAITO inference deployments at any time. 

Learn more.