Search results

Dev Scraping APIs

github.com

Dev Scraping APIs is a GitHub repository or organization with source code, releases, documentation, or project resources.

Awesome Web Scraping

github.com

Awesome Web Scraping is an open source network utility for diagnostics, connectivity checks, or internet testing.

brozzler

github.com

brozzler is an open source project with code, documentation, and setup resources for web archiving and scraping.

datahoarder-website-to-markdown

github.com

datahoarder-website-to-markdown is an open source project with code, documentation, and setup resources for web archiving and scraping.

Dorks Eye Google Hacking Dork Scraping and Searching Script. Dorks Eye is a script I made in python 3. With this tool, you can easily find Google Dorks. Dork Eye collects potentially vulnerable web pages and applications on the Internet or other awesome info that is picked up by Google's search bots. Author: Jolanda de Koff

DownloadNet (dn)

github.com

DownloadNet (dn) is an open source project with code, documentation, and setup resources for web archiving and scraping.

FlareSolverr

github.com

Proxy service that helps scraping tools handle Cloudflare and browser-based anti-bot challenges.

A Playwright-based Node.js tool that bypasses search engine anti-scraping mechanisms to execute Google searches. Local alternative to SERP APIs with MCP server integration.

grab-site

github.com

grab-site is an open source web scraping or crawling project for collecting website data.

Heritrix3

github.com

Heritrix3 is an open source project with code, documentation, and setup resources for web archiving and scraping.

Httrack

github.com

Httrack is an open source project with code, documentation, and setup resources for web archiving and scraping.

MarkSnip is an open source project with code, documentation, and setup resources for web archiving and scraping.

Project Eyes On is a high-speed, multi-threaded surveillance tool by Y0oshi (@rde0) for locating open IP cameras worldwide. Unifies Google Dorking and Directory Scraping into a single OSINT engine.

Resurrect Pages Fork

github.com

Resurrect Pages Fork is an open source project with code, documentation, and setup resources for web archiving and scraping.

Scoop

github.com

Scoop is an open source project with code, documentation, and setup resources for web archiving and scraping.

Scrapling

github.com

Scrapling is an open source web scraping or crawling project for collecting website data.

Kotlin/Java library and cli tool for scraping posts and media from various sources with neither authorization nor full page rendering (Facebook, Instagram, Twitter, Youtube, Tiktok, Telegram, Twitch, Reddit, 9GAG, Pinterest, Flickr, Tumblr, Coub, Vimeo, IFunny, VK, Odnoklassniki, Pikabu)

Spider Suite

github.com

Spider Suite is an open source web scraping or crawling project for collecting website data.

SuckIT

github.com

SuckIT is an open source project with code, documentation, and setup resources for web archiving and scraping.

Trawl

github.com

Trawl is an open source project with code, documentation, and setup resources for web archiving and scraping.

全自动短视频搬运工具，支持自动下载、去重、AI生成标题+标签、上传，可二开扩展至多平台，例如：TikTok->视频号/抖音/小红书、抖音->TikTok/视频号/小红书......video-processing, automation, tiktok, selenium, pyqt5, ffmpeg, bot, data-scraping, video-deduplication.

Waymore

github.com

Waymore is an open source project with code, documentation, and setup resources for web archiving and scraping.

Social Media Hacker List is a code project with source, releases, documentation, or setup notes.

) - Nostr will fail if it stays just another social media protocol. This repo lists ideas for non-social-media applications.