Directory
Published directory entries matching your search.
Dizzy is an open-source crawling and analysis system for onion services. It implements techniques to explore, update, check, and classify onion services at scale without overwhelming the Tor network. It was deployed in April 2021 and used to analyze more than 63.3 million crawled onion webpages, focusing on domain operations, web content, cryptocurrency usage, and web graph.
80legs offers customizable web scraping, including custom web crawling run on a cloud-enabled platform and a Giant Web Crawl product that pulls custom data from a crawl of the entire web. It also points to Datafiniti for instant access to web data.
specification.website is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
WAIL (Web Archiving Integration Layer) is a desktop application that provides a graphical user interface atop multiple pre-configured web archiving tools, intended as an easy way to preserve and replay web pages. It includes Heritrix 3.7.0 for web crawling and OpenWayback 2.4.0 for replaying web archives, is written in Python and compiled with PyInstaller, and is licensed under MIT.
Mwmbl is an open source, non-profit web search engine that operates without ads, tracking or profit motive, and aims to serve as a replacement for commercial search engines. The project uses a centralized index with distributed volunteer-run web crawling, and community members determine search result rankings. Volunteers can contribute by running a command line crawler or installing a Firefox extension to help evaluate search ranking performance. Official documentation is available at https://book.mwmbl.org.
Octopus.do is a visual sitemap and website planning tool. It offers a visual sitemap builder with lo-fi wireframes for prototyping website architecture, and allows crawling a website, choosing a template, or generating with AI. Sitemaps can be shared without sign-up, and exports include PNG, PDF, Excel, Figma, and WordPress.
1000UserGuide is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
advertools is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
Agentic-SEO-Skill is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
aimeos is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
aimeos-typo3 is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
anqicms is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
ArchiveWeb.page is a browser extension and standalone application that captures web archives interactively while browsing. Archived pages can be viewed using ReplayWeb.page without an extension. Automated whole-website crawling is handled by separate tools such as Browsertrix.
astro-i18next is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
astro-seo is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
awesome-seo is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
Black SEO Analyzer is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
client-side-rendering is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
dotplant2 is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
ecommercetools is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
elderjs is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
elm-pages is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
google-indexing-script is an open source tool or library for SEO, metadata, crawling, routing, or website optimization.
grab-site is a web crawler developed by ArchiveTeam for web archiving. It outputs WARC files, offers a dashboard for monitoring all crawls, and supports dynamic ignore patterns to customize crawling behavior.
