Directory
Published directory entries matching your search.
Web Scraping FYI is a knowledge base covering popular technology, concepts and techniques used in web scraping. Its sections address web scraping as a subject, a primer on how the modern web works, other tools and communities, and popular community libraries. Pages are designed to be self-sufficient and indexable.
A GitHub repository listing packages, services and manuals related to web scraping. It covers libraries for Python, PHP, Ruby, JavaScript and Go, command line tools, and web-scraping manuals. It also lists headless browsers, DNS over HTTPS providers, pastebin sites, tunneling software and captcha solving services.
Dev Scraping APIs is a GitHub repository or organization with source code, releases, documentation, or project resources.
Airtable-based OSINT resource list for simple scraping workflows and Twitter-focused research links.
Wget2 is an open source project with code, documentation, and setup resources for web archiving and scraping.
Arch Tools provides one API key for 63 tools aimed at AI agents and developers, covering web search, scraping, AI generation, OCR, transcription, crypto data, email and utilities. It supports REST and MCP protocols and offers two authentication methods: an API key or per-call x402 crypto payments in USDC on Base.
SearchTweaks is a website offering tools to improve web search experiences, specifically enhancing Google search functionality. Its tools are organized into four categories: Query Builders, News-Related Search, Time-Related Search and Search Utilities. Some tools simplify existing Google features, while others add new search capabilities. The site requires JavaScript to function, is designed for desktop use, and does not rely on the Google API or web scraping. It uses Simple Analytics for privacy-focused usage tracking.
Digital Methods tool for scraping search engine results and collecting web research data.
Proxy service that helps scraping tools handle Cloudflare and browser-based anti-bot challenges.
Import.io is a web data extraction and AI infrastructure platform. It converts website content into structured data, and provides web scraping tools, managed data services, pricing intelligence, and MCP tools for AI agents. It supports use cases across multiple industries including retail, ecommerce and AI development.
Browser extension for Instant Data Scraper, adding useful tools or workflow improvements inside the browser.
Ui.Vision RPA is an open-source browser and desktop automation tool for building reusable macros. It offers web automation, desktop automation, web scraping, UI testing and test automation, using visual recording, Selenium-style commands or a JavaScript API. The V10 release adds an AI macro assistant and an MCP bridge for connecting AI assistants. The browser extension is available for Chrome, Edge and Firefox.
80legs offers customizable web scraping, including custom web crawling run on a cloud-enabled platform and a Giant Web Crawl product that pulls custom data from a crawl of the entire web. It also points to Datafiniti for instant access to web data.
Browserflow is a browser-based tool for scraping websites and automating tasks on any website without coding. It offers a visual editor, action recording and custom JavaScript, and flows can run locally in the browser or be deployed to the cloud with scheduling and Google Sheets integration.
brozzler is an open source project with code, documentation, and setup resources for web archiving and scraping.
CopySite helps crawl, scrape, copy, or extract data and pages from websites.
Cyber Gateway presents a free proxy list with HTTP, HTTPS and SOCKS4/5 proxies, described as IP addresses scraped from the Internet. The site also lists stable proxy servers, which it says support tasks such as parsing, scraping, multi-accounting, trading and price monitoring, from a pool said to comprise 75,000 IPs.
Cyotek WebCopy is a free Windows tool that scans a specified website and downloads its content for local offline browsing. It follows links to pages, images, videos and other resources, remapping them to local paths. Configuration includes rules, form and password authentication, and reports. It does not include a virtual DOM or JavaScript parsing, so JavaScript-heavy sites may not copy accurately.
Data Hoarding is an index of resources and archives related to data hoarding, web archival and digital preservation. It provides a list of resources for getting started, helping archival teams, or backing up web content for personal use, with entries like Archive Team, ArchiveBox, Archivematica and ArchivesSpace.
datahoarder-website-to-markdown is an open source project with code, documentation, and setup resources for web archiving and scraping.
DownloadNet (dn) is an open source project with code, documentation, and setup resources for web archiving and scraping.
FreshRSS is a free, self-hostable RSS and Atom feed aggregator released under AGPL 3. It follows websites, podcasts and video channels in one place, supports reading articles directly, saved searches, web scraping for feed generation and OPML import and export. It is translated into more than 15 languages and works with or without a third-party mobile application.
Heritrix is a web archiving or scraping tool for saving pages, data, or online content.
Httrack is an open source project with code, documentation, and setup resources for web archiving and scraping.
