Directory
Published directory entries matching your search.
Quilt is a scientific data management platform hosted on Amazon Web Services (AWS) that helps teams and AI systems locate, verify the trustworthiness of, and reuse data via deeply versioned, context-rich data packages.
The Data Science Lifecycle Process is a repeatable, sustainable process for guiding data science teams from initial idea to delivered value. The process is fully documented in the public GitHub repository dslp/dslp.
DataDeps.jl is a Julia package designed to facilitate reproducible data setup for scientific research. It manages data dependencies and includes features such as progress callbacks. The repository is maintained on GitHub.
DataFrames.jl is a Julia library that provides in-memory tabular data structures for data manipulation and analysis. It is developed by the JuliaData organization and its source code is hosted on GitHub.
Deepnote is a drop-in replacement for Jupyter with an AI-first design. It uses the Deepnote kernel, adds block types, native data integrations and an AI agent, and supports Python, R and SQL locally in an IDE before scaling to Deepnote Cloud for collaboration and deployable data apps.
Data Version Control - Git for Data & Models - ML Experiments Management.
Gopherdata's tooling directory is a curated list of tools, packages, and libraries for doing data science with Go. It organizes resources across categories such as classification, clustering, CSV handling, distributed data analysis, geospatial work, machine learning, and plotting, and also lists proposed packages meant to fill gaps in the ecosystem.
Impala is a distributed C++ query engine for SQL queries over petabytes of data stored in Apache Hadoop clusters. It supports data in HDFS, Apache HBase and Amazon S3, window functions and subqueries, and uses LLVM for on-the-fly code generation. It is Apache-licensed and open source, and the repository is a mirror of Apache Impala.
lakeFS is a source-available data version control tool that turns object storage into a Git-like repository, allowing data lakes to be managed like code. It supports AWS S3, Azure Blob Storage and Google Cloud Storage, is API compatible with S3, and works with frameworks such as Spark, Hive, AWS Athena, DuckDB and Presto.
Onyx is a distributed, masterless data processing system written in pure Clojure, with a batch and stream hybrid processing model. The repository is archived and read-only. Listed uses include realtime event stream processing, extract, transform, load, and data cleaning.
LLM Comparator is an interactive data visualization tool for evaluating and analyzing large language model responses side-by-side, developed by the PAIR team. It includes a Python library for creating JSON files and a defined JSON data format.
PipeRider is a tool described as code review for data in dbt. It compares data to highlight differences in impacted downstream dbt models, profiles dbt models, and generates a static HTML report each run. The project states that PipeRider has been superseded by Recce and will no longer be updated on a regular basis.
PyCM is a multi-class confusion matrix library written in Python. It supports input data vectors and direct matrices and offers post-classification model evaluation statistics. It is aimed at data scientists needing metrics for predictive models and classifiers. A MATLAB setup is also documented.
The repository holds materials and IPython notebooks for "Python for Data Analysis, 3rd Edition" by Wes McKinney, published by O'Reilly Media. Notebooks cover chapters on Python basics, NumPy, pandas, data loading, cleaning, visualization, time series and modeling libraries, plus appendices. The code is released under the MIT license.
Skale is a distributed data processing and machine learning engine that provides a high-level API in JavaScript and a parallel execution engine on top of Node.js. The repository was archived by its owner on May 29, 2021 and is read-only. It supports multiple data sources and formats, and can run in local or distributed mode.
Snorkel is a system for quickly generating training data with weak supervision, released under an Apache-2.0 license and written in Python. The project started at Stanford in 2015 to programmatically label, build, and manage training data. The team now focuses on Snorkel Flow, an end-to-end AI application development platform based on the core ideas behind Snorkel.
Tonic is a tool for downloading, manipulating and loading event-based or spike-based data, described as similar to PyTorch Vision but for neuromorphic data. It provides event datasets and transformations, and the project states it is looking for a new maintainer.
Upgini is an open-source data search and enrichment library for Machine Learning and AI pipelines. It enables users to find and add relevant features from hundreds of public and premium external data sources, including open and commercial large language models.
Open CC-BY-4.0 dataset of evidence-graded dietary-supplement data covering dosing, bioavailability, drug-nutrient interactions and cost-per-effective-dose, with every clinical claim tied to a PubMed PMID. Compiled by Verified Supplement Data, it is mirrored on Hugging Face and Kaggle and offered as a free live JSON API.
AI Models Catalog is a structured YAML catalog of AI model providers and their models, sourced from first-party APIs and official documentation. It includes TypeScript type definitions and Zod runtime validation schemas, plus scripts to sync data from providers and validate YAML files against schemas.
CPU and GPU-accelerated matrix library intended to support large-scale exploratory data analysis.
NumPy and Pandas interface to Big Data.
Bowtie is a library for writing dashboards in Python. It wraps the code needed for interactive data exploration and visualization, so users can build functionality without knowing web frameworks or JavaScript. It can be installed with conda or pip, and Docker images are provided.
Cerberus is a lightweight and extensible data validation library for Python. It provides type checking and other base functionality out of the box, is designed to be non-blocking and extensible for custom validation, and has no dependencies. It is available on PyPI and maintained according to semantic versioning since version 1.2.
