LLMKube

LLMKube is a Kubernetes operator for self-hosted large language model inference across heterogeneous GPU fleets supporting NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. It supports runtimes including llama.cpp, vLLM, TGI, and mlx-server, offers multi-GPU sharding, model caching, and OpenAI-compatible endpoints. It is licensed under Apache-2.0 and designed for homelab and on-prem fleet deployment, and is actively developed.

GitHub repo Directory import
0 shouts Added Sep 23, 2026 github.com
github.com/defilantech/LLMKube
LLMKube website preview

Links

Tags