Directory
Published directory entries matching your search.
BROWSEO is a web app for viewing web pages without distracting styles and with SEO-relevant parts highlighted. It requires no download or installation; a URL is entered to start. The output focuses on pure HTML similar to what search engines see, with toggles for elements such as server response code, headings, links and META information.
claude-skills crawls websites to audit SEO, accessibility, performance, and technical page issues.
Clicks.so helps audit websites, improve search visibility, and monitor SEO or analytics performance.
CopySite helps crawl, scrape, copy, or extract data and pages from websites.
CoVaxxy helps audit websites, improve search visibility, and monitor SEO or analytics performance.
Open source web crawler built for extracting clean website data for AI and LLM workflows.
CrawlGraph is a backlink and competitor-gap tool built on Common Crawl snapshots. It returns referring domains with authority fields and compares competitors through a link-gap endpoint. It does not run a live daily crawler. The free website query returns five referring domains without signup; paid plans unlock full lists, CSV/JSON export, gap analysis and API access.
crawlie crawls websites to audit SEO, accessibility, performance, and technical page issues.
Crawly is a web crawler by Diffbot that spiders and extracts complete structured data from an entire website. It automatically extracts the correct content from any web page without rules, and extracted fields can be downloaded in CSV or as JSON.
Cyotek WebCopy is a free Windows tool that scans a specified website and downloads its content for local offline browsing. It follows links to pages, images, videos and other resources, remapping them to local paths. Configuration includes rules, form and password authentication, and reports. It does not include a virtual DOM or JavaScript parsing, so JavaScript-heavy sites may not copy accurately.
DAXRM helps audit websites, improve search visibility, and monitor SEO or analytics performance.
Fess is an open-source, self-hosted enterprise and site search server built on OpenSearch, installable on any platform with a Java runtime. A built-in crawler collects documents from web sites, file systems, and data stores such as databases and CSV files. It is configured through a browser-based administration UI and offers a REST API.
Focsec is an API for detecting VPNs, proxies, TOR and bot traffic. It provides IP check data for protecting applications, identifying suspicious logins, detecting fraudulent signups and stopping scrapers and crawlers. Plans are billed monthly, and a free 14 day trial is offered.
geo-optimizer-skill crawls websites to audit SEO, accessibility, performance, and technical page issues.
grab-site is a web crawler developed by ArchiveTeam for web archiving. It outputs WARC files, offers a dashboard for monitoring all crawls, and supports dynamic ignore patterns to customize crawling behavior.
GrackerAI helps audit websites, improve search visibility, and monitor SEO or analytics performance.
Hashatit helps audit websites, improve search visibility, and monitor SEO or analytics performance.
Heritrix is the Internet Archive's open-source web crawler project, described as extensible, web-scale and archival-quality. It is designed to respect robots.txt exclusion directives and META nofollow tags. Documentation covers getting started, operating, configuring crawl jobs and the REST API. It is free software under the Apache License, Version 2.0, with some files under other licenses.
Liveuamap is a map-centric news site founded in 2014 by developers and journalists. Liveuamap shows conflict and other news geographically, using AI web crawlers and analysts to fact-check stories before editors place them on the map. The site offers an archive of past site information and has expanded its coverage beyond the Ukrainian conflict to other regions, including Israel-Palestine.
Generate and download e-books from online sources.
Mangools is a suite of SEO tools for keyword research, SERP analysis, rank tracking, backlink analysis and competitor research. Its tools include KWFinder, SERPChecker, SERPWatcher, LinkMiner and SiteProfiler, with a keyword database of over 2.5 billion keywords. Rank tracking is weekly, with daily tracking on the Agency plan.
Market Muse helps audit websites, improve search visibility, and monitor SEO or analytics performance.
mcp-gsc helps monitor indexing, search performance, crawl data, and site visibility.
Mentionmapp Analytics helps audit websites, improve search visibility, and monitor SEO or analytics performance.
