github.com
Text Generation Inference is a toolkit for deploying and serving large language models, implemented as a Rust, Python and gRPC server. It supports models including Llama, Falcon, StarCoder, BLOOM and GPT-NeoX, with features such as token streaming, continuous batching and quantization. The repository is archived, read-only, and in maintenance mode, with pull requests limited to minor bug fixes and maintenance.