Directory
Published directory entries matching your search.
lakeFS is a source-available data version control tool that turns object storage into a Git-like repository, allowing data lakes to be managed like code. It supports AWS S3, Azure Blob Storage and Google Cloud Storage, is API compatible with S3, and works with frameworks such as Spark, Hive, AWS Athena, DuckDB and Presto.
Modern columnar data format for ML implemented in Rust.
LightlyTrain is an open-source GitHub repository offering all-in-one training capabilities for vision models, including YOLO, ViTs, RT-DETR, and DINOv3. It supports pretraining, fine-tuning, and distillation workflows for these model types.
Luigi is a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures and command line integration, and includes Hadoop support. It is intended to stitch together long-running tasks such as Hive, Pig, Hadoop, Spark or database jobs.
Lux is a Python library for automated data exploration. When a pandas dataframe is printed in a Jupyter notebook, Lux recommends a set of visualizations highlighting trends and patterns, shown in an interactive widget for browsing and exploring the dataset.
A federated, open-source data catalog for all your big data and small data.
Marquez is an open source metadata service for collecting, aggregating and visualizing a data ecosystem's metadata. It maintains provenance of how datasets are consumed and produced and provides visibility into job runtime and dataset access. Metadata is collected via OpenLineage, and a web UI shows job and dataset dependencies through a lineage graph.
Mars is an open source tensor-based unified framework for large-scale data computation. It scales numpy, pandas and scikit-learn, offering tensor, DataFrame and learn interfaces that resemble the original libraries. A session can be started locally or connected to an existing Mars cluster.
MeTA is a modern C++ data sciences toolkit. It provides text tokenization including deep semantic features like parse trees, inverted and forward indexes with compression and caching strategies, ranking functions for searching indexes, topic models, classification algorithms, graph algorithms, language models, a CRF implementation for POS-tagging and shallow parsing, and wrappers for liblinear and libsvm.
Unified metadata exploration API service for Hive, RDS, Teradata, Redshift, S3 and Cassandra.
Analyzes structured data and helps produce queries, insights or visualizations.
ModelDB is an open-source system for versioning machine learning models, including their code, data, config and environment, and for tracking ML metadata across the model lifecycle. It supports reproducible models, experiment management and lifecycle tracking. It runs on Docker and Kubernetes, offers Python and Scala clients, and is maintained by Verta.ai.
Algorithm capable of fully capturing the impact of data drift on performance.
Notebooks is a Docker-based starter kit for machine learning through Jupyter notebooks, intended for users who want a ready runtime environment. Its images cover combinations of Keras, TensorFlow and PyTorch with CUDA v8 through v11, and include JupyterLab, scikit-learn, pandas and visualization, vision and NLP libraries. Python 3 only, as Python 2 is end-of-life.
notespace is an open source project hosted as a public repository on GitHub under the scicloj/notespace project. It is designed to enable users to use their namespace as a notebook. The project invites contributions from developers.
Onyx is a distributed, masterless data processing system written in pure Clojure, with a batch and stream hybrid processing model. The repository is archived and read-only. Listed uses include realtime event stream processing, extract, transform, load, and data cleaning.
MinerU is a tool that transforms complex documents such as PDFs and Office documents into LLM-ready markdown or JSON for agentic workflows. It supports multiple input formats including PDF, images, DOC/DOCX, PPT/PPTX, XLS/XLSX, RTF, ODT/ODS/ODP, EPUB, OFD, HTML/MHTML and CSV/TSV. Version 4.0 offers four parsing tiers and a document library with search and citation locators, and documents are not automatically uploaded to the official service; remote parsing requires explicit configuration.
Bring multiple data streams into one dashboard.
Cleansing, pre-processing, feature engineering, exploratory data analysis and easy ML with PySpark backend.
Orkas is an open-source, local-first AI desktop app. A commander LLM directs specialist sub-agents and runs installed coding CLIs including Claude Code, Codex, OpenCode, OpenClaw, and Hermes as local sessions. Its agents self-evolve via reflection and skill crystallization, supports bring-your-own keys, and is compatible with macOS, Windows, and Linux.
Pachyderm is a project for data-centric pipelines and data versioning. It provides parallelized processing of multi-stage, language-agnostic pipelines with data versioning and data lineage tracking, and pipelines trigger automatically when data changes. It runs on Kubernetes and uses standard object stores for storage.
Create HTML profiling reports from pandas DataFrame objects.
Panthera is an open-source Clojure library available on GitHub that provides data-frame and array functionality for the Clojure programming language. The repository is maintained by alanmarazzi and accepts contributions from users.
ParaMonte is a serial and parallel library of Monte Carlo routines for sampling mathematical density functions of arbitrary dimensions and machine learning algorithms for scientific inference. It is designed for interoperability with C, C++, Fortran, MATLAB and Python, and supports shared-memory OpenMP threads and MPI distributed communications.
