github.com
Inference engine for TensorRT on Nvidia GPUs.
Directory
Search results
Published directory entries matching your search.
Filters
github.com
A toolkit for deploying and serving open source text embeddings and sequence classification models. It supports models including FlagEmbedding, Ember, GTE and E5, and features token based dynamic batching, Flash Attention and Candle optimized inference, Safetensors and ONNX weight loading, and Metal support for local execution on Macs.
github.com
Text Generation Inference is a toolkit for deploying and serving large language models, implemented as a Rust, Python and gRPC server. It supports models including Llama, Falcon, StarCoder, BLOOM and GPT-NeoX, with features such as token streaming, continuous batching and quantization. The repository is archived, read-only, and in maintenance mode, with pull requests limited to minor bug fixes and maintenance.
github.com
Flexible, high-performance serving system for machine learning models.
github.com
TNN is a deep learning inference framework for mobile, desktop and server, developed by Tencent Youtu Lab and Guangying Lab. It is based on ncnn and Rapidnet and extends support to X86 and NV GPUs. The project states that it has been used in Tencent applications including Mobile QQ, Weishi and Pitu.
github.com
TorchServe is an open-source tool for serving, optimizing, and scaling PyTorch models in production. It offers quick start guides for TorchServe, conda, and Docker, and includes an announcement about security changes.
github.com
Traceloop OpenLLMetry is an open-source observability tool for GenAI and LLM applications, based on OpenTelemetry and maintained by Traceloop. It provides instrumentation for vector databases and frameworks, and includes documentation on getting started and supported destinations.
github.com
Triton Inference Server is open source inference serving software that streamlines AI inferencing. It enables teams to deploy models from multiple deep learning and machine learning frameworks across cloud, data center, edge and embedded devices.
github.com
Vald is a highly scalable distributed vector search engine. It is hosted on GitHub and provides documentation on installation using Helm, requirements, and usage. The project focuses on delivering a scalable solution for vector search.
github.com
Scalable, fast, and disk-friendly vector search in Postgres, the successor of pgvecto.rs.
github.com
VectorDB is a Python vector database designed to be simple and sufficient. It can be used locally or as a service, with server and client components. The project emphasizes ease of use for vector database needs.
github.com
Vespa is an AI search platform. Its repository contains the code required to build and run Vespa. The project covers searching, making inferences in and organizing vectors, tensors, text and structured data at serving time and any scale. It is licensed under Apache 2.0.
github.com
Open-source project that supports deploying, serving, monitoring, or operating AI and machine-learning systems.
github.com
Weights & Biases is an open-source AI developer platform. It is used to train and fine-tune machine learning models, and manage models across the full lifecycle from experimentation to production. The associated wandb library is hosted in a public GitHub repository.
github.com
Whylogs is an open-source data logging library for machine learning models and data pipelines. It provides visibility into data quality and model performance over time, and supports privacy-preserving data collection to support system safety and robustness.
github.com
Xinference is a model serving tool that provides a unified production-ready inference API. It allows users to run open-source, speech and multimodal models on cloud infrastructure, on-premises systems or personal laptops by modifying a single line of code.
github.com
xTuring is an open-source project for building, personalizing, and controlling LLMs, covering data pre-processing through fine-tuning. It includes a simple API for data prep, training, and inference, LoRA and low-precision (INT8/INT4) support, and built-in evaluation metrics such as perplexity. Models can run locally or in a private VPC and scale from CPU to multi-GPU.
github.com
ZenML is an open-source AI platform for building workflows that run on various infrastructure backends. It allows writing pipelines containing Python logic, such as model training or agentic loops, and tracks runs with metrics, logs and metadata. It integrates with tools and infrastructure including MLflow, LangGraph, Langfuse, Sagemaker and GCP Vertex.