Directory
Published directory entries matching your search.
Heritrix is the Internet Archive's open-source web crawler project, described as extensible, web-scale and archival-quality. It is designed to respect robots.txt exclusion directives and META nofollow tags. Documentation covers getting started, operating, configuring crawl jobs and the REST API. It is free software under the Apache License, Version 2.0, with some files under other licenses.
Httrack is an open source project with code, documentation, and setup resources for web archiving and scraping.
Infinity New Tab is a new tab page extension based on Chrome's new tab page. It offers management of frequently accessed websites with redrawn icons, one-click addition of sites, swift switching among search engines, and customizable wallpapers. It also includes sidebar gadgets. Installation is available on Chrome with an offline Crx file option.
Irchiver is a research project and software that automatically takes screenshots of web content and provides full-text search of browsing history. It runs on Windows and Mac with several browsers and stores screenshots as WebP plus text files in a local folder without network connectivity.
MarkdownDown is a web archiving or scraping tool for saving pages, data, or online content.
MarkSnip is an open source project with code, documentation, and setup resources for web archiving and scraping.
Packet Sender is a free, open source utility for sending and receiving network packets, with support for TCP, UDP, SSL and DTLS. It includes a built-in HTTP client for GET/POST requests, panel generation, a command line interface for scripting, and runs on Windows, Mac and Linux.
PageRip is a Chrome extension that extracts HTML and CSS from web pages using an element selector, or converts full pages to single-file HTML and JSX. It can export to CSS classes or inline styles, and code can be previewed and copied. The tool is described as completely free and works offline.
Pinging.net is a web tool for testing and monitoring an internet connection. It reports status for HTTP, DNS, WebRTC and the browser, and shows web ping statistics including last, minimum, maximum and average values and packet loss, with updates every 30 seconds.
PingTracer is an open source project with code, documentation, and setup resources for network diagnostics.
wireshark profiles provides Wireshark profiles or network analysis resources for packet inspection workflows.
A reference page of links for searching cached copies and web archives. It advises opening all links in new tabs, noting this can slow down or suspend the browser, and suggests refreshing if archives fail to load. It also recommends searching for the site name with a site:public4.pagefreezer.com query or trying PageFreezer when no copy is found.
ReplayWeb is a web archiving or scraping tool for saving pages, data, or online content.
Resurrect Pages Fork is an open source project with code, documentation, and setup resources for web archiving and scraping.
Scoop is an open source project with code, documentation, and setup resources for web archiving and scraping.
Scrapling is an adaptive web scraping framework that supports tasks ranging from a single request to full-scale crawling. The project offers community channels on Discord and Twitter for tips and support.
SoundFixer is a WebExtension for Firefox that addresses sound problems on the web, such as audio in one channel only, volume too quiet at maximum or too loud at minimum. It is distributed through addons.mozilla.org. It does not work on all websites because the Web Audio API cannot be used from a cross-domain source.
Speedtest is an open source project with code, documentation, and setup resources for network diagnostics.
Spider Suite is an open source web scraping or crawling project for collecting website data.
SuckIT is an open source project with code, documentation, and setup resources for web archiving and scraping.
Trawl is a self-hosted scraping engine designed to bypass JavaScript challenges and CAPTCHAs including Cloudflare, Turnstile, reCAPTCHA, hCaptcha and GeeTest. It is positioned as an alternative to FlareSolverr and Byparr, and can serve as a drop-in replacement for *arr stack applications.
Vandal is a browser extension that serves as a navigator for the Web Archive, helping users move back in time without leaving the current tab. It is built on the Wayback Machine API and offers calendar and graph views, a built-in browser for archived snapshots, navigation history, a resource drawer and a beta historical view. Limitations include iframe restrictions on some websites and no support for saving pages.
WAIL (Web Archiving Integration Layer) is a desktop application that provides a graphical user interface atop multiple pre-configured web archiving tools, intended as an easy way to preserve and replay web pages. It includes Heritrix 3.7.0 for web crawling and OpenWayback 2.4.0 for replaying web archives, is written in Python and compiled with PyInstaller, and is licensed under MIT.
Waymore gathers links from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal, GhostArchive and Intelligence X. It can also download archived responses for URLs on the Wayback Machine so those can be searched for more links, developer comments and extra parameters. The tool is intended for coverage rather than speed.
