Heritrix3

Heritrix is the Internet Archive's open-source web crawler project, described as extensible, web-scale and archival-quality. It is designed to respect robots.txt exclusion directives and META nofollow tags. Documentation covers getting started, operating, configuring crawl jobs and the REST API. It is free software under the Apache License, Version 2.0, with some files under other licenses.

GitHub repo Directory import
0 shouts Added Sep 23, 2026 github.com
github.com/internetarchive/heritrix3
Heritrix3 website preview

Features

  • designed to respect the robots.txt exclusion directives Source: github.com

Links

Tags