github.com
siteone-crawler checks page speed, performance, technical issues, and web quality signals.
Directory
Search results
Published directory entries matching your search.
Filters
github.com
Light Novel Crawler Colab is a code project with source, releases, documentation, or setup notes.
colab.research.google.com
LN-Crawler-Colab helps find, read, track, or manage fiction, fanfiction, web novels, and stories.
usenet-crawler.com
usenet-crawler is a software download or discovery resource for apps, installers, releases, or tools.
github.com
Lightnovel Crawler is a code project with source, releases, documentation, or setup notes.
monstercrawler.com
Monster Crawler Search is a web search engine that aggregates results from top authority sites and major search engines including Google and Yahoo! to deliver search results. It supports web, image, video and news search categories.
arxiv.org
CRATOR is a proposed dark web crawler described in an arXiv paper. It is designed to extract pages that handle security protocols such as captchas, using seed URL lists, link analysis and scanning to discover content, with user-agent rotation and proxy usage for anonymity. The authors report evaluation using coverage, performance and robustness metrics, and suggest threat intelligence, cybersecurity and online investigation applications.
domaincrawler.com
Domain Crawler is a web tool for finding, checking, or analyzing online information related to domains, networks, and digital services.
iocaine.madhouse-project.org
Iocaine is an aggressive defense mechanism that tries to poison AI crawlers and other unwelcome bots by serving them garbage, rather than making them go away. It is designed to be lightweight and efficient, spending as few resources as possible while making the life of crawlers more expensive. It is intended for use together with a reverse proxy.
librecrawl.com
LibreCrawl is a free, open-source SEO crawler offered as an alternative to Screaming Frog. It performs technical audits covering meta tags, schema markup, hreflang, social tags and PageSpeed metrics, and integrates Playwright for JavaScript rendering of single-page applications. The project is MIT-licensed and can be self-hosted, with CSV, JSON and XML exports. A REST API provides programmatic control of crawls.
screamingfrog.co.uk
Screaming Frog SEO Spider is a website crawler for Windows, macOS and Ubuntu used for technical SEO site audits. It audits sites for over 300 SEO issues and can find broken links, audit redirects, analyse page titles and meta data, and generate XML sitemaps. A free version is available.
site-analyzer.pro
SiteAnalyzer is a website crawler for technical audits and SEO optimization. It scans site pages and checks more than 60 parameters, including broken links, duplicate pages, meta tags and server response codes. It offers integrations with Google Search Console, Yandex Webmaster and Yandex Metrica, internal PageRank calculation, site structure visualization and sitemap generation. A free version and a Pro plan are available.
sitebulb.com
Sitebulb is a website crawler for SEO audits, available as desktop software and a cloud service. It reports prioritized hints, audit comparison and JavaScript crawling. Desktop crawls up to 500,000 URLs per audit; cloud supports up to 10m URLs per audit with real-time team collaboration.
github.com
Open-source research crawler for collecting links and metadata from reachable onion pages.
ads.google.com
Anonymous Ad Preview Tool helps audit websites, improve search visibility, and monitor SEO or analytics performance.
search.google.com
Google Search Console helps monitor indexing, search performance, crawl data, and site visibility.
pypi.org
Python OSINT crawler for extracting URLs, files, endpoints, and related website data.
github.com
aaron-marketing-skills crawls websites to audit SEO, accessibility, performance, and technical page issues.
ahmia.fi
Ahmia is a search engine for services on the Tor network. It indexes hidden services, which require the Tor browser bundle to access, and states that abuse material is not allowed and is removed from the index. The project is developed as free and open source software, with a crawler, an index and a search website.
github.com
ai-marketing-claude crawls websites to audit SEO, accessibility, performance, and technical page issues.
alttextlab.com
AltTextLab is an AI-powered alt text generator for images. It produces descriptions in 130+ languages and integrates with WordPress, Shopify, Webflow, and 20+ tools, also offering a web snippet and REST API. It provides 25 free credits monthly with no credit card required.
crawler.amethyst.social
Find and broadcast nostr events
bishopi.io
Bishopi.io helps audit websites, improve search visibility, and monitor SEO or analytics performance.
bitmagnet.io
BitMagnet is a self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with a web UI, GraphQL API and Servarr stack integration. The software is currently in alpha, and the documentation notes likely bugs plus API and database schema changes before a theoretical 1.0 release.