GPT-NeoX

GPT-NeoX is EleutherAI's library for training large-scale language models on GPUs. It builds on NVIDIA's Megatron and adds DeepSpeed techniques plus optimizations. It supports launching via Slurm, MPI and the IBM Job Step Manager, and offers predefined configurations for architectures including Pythia, PaLM, Falcon and LLaMA 1 & 2. The repository states it is likely the wrong library for those not training models with billions of parameters from scratch.

GitHub repo Directory import
0 shouts Added Sep 23, 2026 github.com
github.com/EleutherAI/gpt-neox
GPT-NeoX website preview

Features

  • launching via Slurm, MPI, and the IBM Job Step Manager Source: github.com

Uses

  • if you are not looking to train models with billions of parameters from scratch Source: github.com

Links

Tags