Directory
Published directory entries matching your search.
80legs offers customizable web scraping, including custom web crawling run on a cloud-enabled platform and a Giant Web Crawl product that pulls custom data from a crawl of the entire web. It also points to Datafiniti for instant access to web data.
A GitHub repository listing packages, services and manuals related to web scraping. It covers libraries for Python, PHP, Ruby, JavaScript and Go, command line tools, and web-scraping manuals. It also lists headless browsers, DNS over HTTPS providers, pastebin sites, tunneling software and captcha solving services.
Betterfox is a user.js configuration file of about:config tweaks intended to enhance Mozilla Firefox privacy and security without third-party code. The project includes the user.js, described as containing all the essentials, plus guides named Securefox, Peskyfox and Smoothfox covering settings within Firefox.
brozzler is an open source project with code, documentation, and setup resources for web archiving and scraping.
CachedView searches multiple providers, including the Wayback Machine, Archive.today and others, to check whether a webpage is in their cache and links to the results. Users enter a URL to receive its current status and view cached or archived results.
ClippingMini is a bookmark manager that saves links, files and documents into folders. The source states that AI tags, categorizes and summarizes them, checks unsafe URLs and previews documents, video and audio in the browser. It is available through a Chrome extension and a web app, and it can be connected in ChatGPT for saving links and searching the library.
Clippings is a browser extension or add-on that improves everyday web browsing workflows.
Wget Wizard is a web form that helps users build Wget command lines through a user interface covering popular options. It does not download files itself; it suggests a command to run with Wget on a computer or server. The page notes advanced options are omitted.
datahoarder-website-to-markdown is an open source project with code, documentation, and setup resources for web archiving and scraping.
DownloadNet (dn) is an open source project with code, documentation, and setup resources for web archiving and scraping.
Proxy service that helps scraping tools handle Cloudflare and browser-based anti-bot challenges.
Globalping is a platform for running network commands such as ping, traceroute and DNS resolve on probes distributed worldwide. It is used to monitor, debug and benchmark internet infrastructure. It offers a REST API, CLI and integrations.
grab-site is a web crawler developed by ArchiveTeam for web archiving. It outputs WARC files, offers a dashboard for monitoring all crawls, and supports dynamic ignore patterns to customize crawling behavior.
Heritrix is a web archiving or scraping tool for saving pages, data, or online content.
Heritrix is the Internet Archive's open-source web crawler project, described as extensible, web-scale and archival-quality. It is designed to respect robots.txt exclusion directives and META nofollow tags. Documentation covers getting started, operating, configuring crawl jobs and the REST API. It is free software under the Apache License, Version 2.0, with some files under other licenses.
Httrack is an open source project with code, documentation, and setup resources for web archiving and scraping.
Infinity New Tab is a new tab page extension based on Chrome's new tab page. It offers management of frequently accessed websites with redrawn icons, one-click addition of sites, swift switching among search engines, and customizable wallpapers. It also includes sidebar gadgets. Installation is available on Chrome with an offline Crx file option.
Irchiver is a research project and software that automatically takes screenshots of web content and provides full-text search of browsing history. It runs on Windows and Mac with several browsers and stores screenshots as WebP plus text files in a local folder without network connectivity.
MarkdownDown is a web archiving or scraping tool for saving pages, data, or online content.
MarkSnip is an open source project with code, documentation, and setup resources for web archiving and scraping.
Monkeytype is a minimalistic, customizable typing test website. It offers multiple test modes, an account system to save typing speed history, and configurable options such as themes, sounds and a smooth caret, with live displays of errors, words per minute and accuracy.
Mumble is a free, open source voice chat application designed for low latency and high quality communication. The project dates back over a decade and documents features for end-users, administrators and hosters, including an extensive permission system, audio recording and positional audio in games. Server administration documentation covers configuration files and permission settings.
Open source voice chat software for low-latency group communication and self-hosted servers.
Packet Sender is a free, open source utility for sending and receiving network packets, with support for TCP, UDP, SSL and DTLS. It includes a built-in HTTP client for GET/POST requests, panel generation, a command line interface for scripting, and runs on Windows, Mac and Linux.
