Directory
Published directory entries matching your search.
Heritrix is the Internet Archive's open-source web crawler project, described as extensible, web-scale and archival-quality. It is designed to respect robots.txt exclusion directives and META nofollow tags. Documentation covers getting started, operating, configuring crawl jobs and the REST API. It is free software under the Apache License, Version 2.0, with some files under other licenses.
Ladder is a self-hosted HTTP web proxy and developer tool for testing and analyzing paywall implementations and content delivery behavior on websites. It can remove or modify CORS and other headers, inject custom code, apply domain-based rulesets and emulate different client types. The project states it does not circumvent advanced access restrictions and may not function on services that actively restrict access.
