github.com
Curated GitHub list of public datasets across science, government, finance, and research topics.
Directory
Search results
Published directory entries matching your search.
github.com
Facets is a project containing two visualizations, Facets Overview and Facets Dive, for understanding and analyzing machine learning datasets. The visualizations are Polymer web components backed by TypeScript that can be embedded in Jupyter notebooks or webpages. Overview produces feature-by-feature statistical analysis and compares statistics across datasets. The repository was archived on Jul 27, 2024 and is now read-only.
github.com
Scripts to load several popular datasets including:.
github.com
GitHub awesome list collecting OSINT tools, datasets, training links, and research resources.
github.com
AIF360, or AI Fairness 360, is a comprehensive toolkit that provides fairness metrics for datasets and machine learning models. It includes explanations for these metrics and algorithms to mitigate bias in data and models. The project is developed by the Trusted-AI community on GitHub.
github.com
Ambrosia is a cross-platform command line tool written in Go for improving text datasets used in machine learning. It offers fast deduplication, filtering, length checking and whitespace trimming, plus an LLM-based filtering option called psort. It ships as a single binary with no dependencies and is pre-1.0 software.
github.com
A GitHub collection of graph classification papers with reference implementations, covering embedding, deep learning, graph kernel and factorization methods. It also links to graph classification benchmark datasets and similar curated collections.
github.com
BELLE, short for Be Everyone's Large Language model Engine, is an open-source Chinese conversational large language model hosted on GitHub by LianjiaTech. The project provides relevant training code and open datasets for the model.
Latest Papers and Datasets on Multimodal Large Language Models, and Their Evaluation.
github.com
Bread Dataset Viewer is a VS Code extension for viewing and handling large datasets. It opens JSONL, Parquet, CSV, TSV, JSON, and Arrow/Feather files by streaming and lazy-loading data, and provides Cards, Table, and Raw views. It includes 16 production LLM tokenizers from GPT, Claude, Llama, and Qwen for chat completion data.
github.com
Open-source Python library for data-centric AI that helps clean data and labels by automatically detecting issues in machine learning datasets. It uses existing models to estimate dataset problems that can be fixed to train better models. It runs on Python 3.10+ and works with any model across classification, regression, segmentation and object detection tasks.
github.com
CML, or Continuous Machine Learning, is an open-source CLI tool for implementing CI/CD with a focus on MLOps. It is used to automate development workflows, including machine provisioning, model training and evaluation, comparing ML experiments across project history, and monitoring changing datasets. It can generate a visual report with results and metrics automatically on every pull request, using GitLab, GitHub, or Bitbucket.
github.com
Awesome-Code-LLM is a curated list of language modeling research for code and other software engineering activities, plus related datasets. The GitHub repository is marked TMLR and links to an arXiv paper.
github.com
Track your datasets, code changes, experimentation history, and models.
github.com
CorpusLoaders is a Julia package that provides a variety of loaders for different natural language processing (NLP) corpora, maintained by JuliaText and hosted on GitHub. It simplifies accessing and loading diverse NLP datasets for research and development use cases.
github.com
Datumbox is an open-source machine learning framework written in Java for rapid development of machine learning and statistical applications. It focuses on including many machine learning algorithms and statistical methods and on handling large datasets. The code is licensed under the Apache License, Version 2.0, and releases are available on Maven Central, including version 0.8.2.
github.com
Detecto is a Python package for building computer vision and object detection models with PyTorch. It supports inference on still images and videos, transfer learning on custom datasets, and serialization of models to files, and is distributed under an MIT license.
github.com
dLine is a command-line calendar tool that brings a calendar directly to the terminal. It can track important dates, add events using APIs, calculate timespans and manage multiple calendars. It offers a dynamic monthly view, a static monthly calendar, a workday calculator and administration of calendar datasets, and asks for a region on first launch to fetch relevant holidays.
github.com
Future AGI is an open-source, self-hostable platform for evaluating, observing, and improving LLM and AI agent applications. It combines tracing, evals, simulations, datasets, a gateway and guardrails on one platform with a feedback loop. The Apache 2.0 core can be self-hosted, and a managed Cloud option is offered.
github.com
A dataset format for creating, storing, and collaborating on AI datasets of any size.
This repository provides a hyperparameter optimization implementation for machine learning algorithms, accompanying a paper on hyperparameter optimization theory and practice. It includes sample code for regression and classification problems using the Boston-Housing and MNIST datasets, covering algorithms such as random forest, SVM, KNN and artificial neural networks.
github.com
A GitHub repository holding the notebooks and code for the book "Introduction to Machine Learning with Python" by Andreas Mueller and Sarah Guido, together with the mglearn library of helper functions for figures and datasets. The book requires scikit-learn 0.20.0, and the code also needs numpy, scipy, matplotlib, pandas and pillow.
github.com
ManifoldLearning.jl is a Julia package for manifold learning and nonlinear dimensionality reduction. It provides methods including Isomap, diffusion maps, locally linear embedding, Hessian eigenmaps, Laplacian eigenmaps, local tangent space alignment and t-SNE. Most methods use k-nearest neighbors to construct local subspace representation, and a custom k-nearest neighbors function is suggested for large datasets.
github.com
Marquez is an open source metadata service for collecting, aggregating and visualizing a data ecosystem's metadata. It maintains provenance of how datasets are consumed and produced and provides visibility into job runtime and dataset access. Metadata is collected via OpenLineage, and a web UI shows job and dataset dependencies through a lineage graph.