An arXiv paper on digging deeper into hidden web databases limited by the top-k output constraint. It presents the GetNext meta-algorithm, which retrieves the next ranked tuple through a restrictive interface without prior knowledge of the ranking function, and algorithms that generate and execute multiple reformulated queries. Theoretical analysis and experiments over synthetic and real-world databases are reported.
An arXiv paper on discovering skylines over hidden web databases whose search interfaces enforce a top-k output constraint. It examines how search interface controls affect discovery cost and develops algorithms for one-ended range, free range and point predicates, combined for mixed interfaces. Experiments over real-world online and offline databases are reported.
This arXiv paper by Dirk Lewandowski and Philipp Mayr provides a critical review of Bergman's 2001 study on the Deep Web. It introduces the concept of the Academic Invisible Web, defined as databases and collections relevant to academia but not searchable by general-purpose internet search engines, and reviews approaches for indexing it. The authors question Bergman's size estimate and suggest collaboration.
A library support page explaining the invisible web, also called the deep web or hidden web. It states that search engines cover about 16% of the internet and advises against relying on them alone, especially for academic information. It lists content search engines cannot find, such as dynamically changing content, password protected resources and sites requiring registration, and suggests web directories and asking a librarian.
This dataset maps the network layer of the Invisible Internet Project (I2P), collected using the SWARM-I2P framework, which deployed I2P routers as mapping agents. It documents over 50,000 nodes with metrics such as bandwidth, latency, and uptime, plus network traffic records and geographic distribution. Node identifiers were anonymized. Data is available in CSV and TXT formats on Zenodo, with mapping scripts on GitHub.
An arXiv paper measuring onion website discovery and Tor users' interests with honeypots. The authors deployed honeypot onion websites in March–April 2025 and seeded links via three channels to observe interaction events. They report that human users almost without exception originated from Ahmia.fi, and that the CSAM-themed honeypot drew markedly higher engagement than the others.
CRATOR is a proposed dark web crawler described in an arXiv paper. It is designed to extract pages that handle security protocols such as captchas, using seed URL lists, link analysis and scanning to discover content, with user-agent rotation and proxy usage for anonymity. The authors report evaluation using coverage, performance and robustness metrics, and suggest threat intelligence, cybersecurity and online investigation applications.
Community notes
Be the first to share how this project worked for you.
Sign in with a verified account to leave a review or join the discussion.
Sign in to reviewKeep exploring
An arXiv paper on discovering skylines over hidden web databases whose search interfaces enforce a top-k output constraint. It examines how search interface controls affect discovery cost and develops algorithms for one-ended range, free range and point predicates, combined for mixed interfaces. Experiments over real-world online and offline databases are reported.
This arXiv paper by Dirk Lewandowski and Philipp Mayr provides a critical review of Bergman's 2001 study on the Deep Web. It introduces the concept of the Academic Invisible Web, defined as databases and collections relevant to academia but not searchable by general-purpose internet search engines, and reviews approaches for indexing it. The authors question Bergman's size estimate and suggest collaboration.
A library support page explaining the invisible web, also called the deep web or hidden web. It states that search engines cover about 16% of the internet and advises against relying on them alone, especially for academic information. It lists content search engines cannot find, such as dynamically changing content, password protected resources and sites requiring registration, and suggests web directories and asking a librarian.
This dataset maps the network layer of the Invisible Internet Project (I2P), collected using the SWARM-I2P framework, which deployed I2P routers as mapping agents. It documents over 50,000 nodes with metrics such as bandwidth, latency, and uptime, plus network traffic records and geographic distribution. Node identifiers were anonymized. Data is available in CSV and TXT formats on Zenodo, with mapping scripts on GitHub.
An arXiv paper measuring onion website discovery and Tor users' interests with honeypots. The authors deployed honeypot onion websites in March–April 2025 and seeded links via three channels to observe interaction events. They report that human users almost without exception originated from Ahmia.fi, and that the CSAM-themed honeypot drew markedly higher engagement than the others.
CRATOR is a proposed dark web crawler described in an arXiv paper. It is designed to extract pages that handle security protocols such as captchas, using seed URL lists, link analysis and scanning to discover content, with user-agent rotation and proxy usage for anonymity. The authors report evaluation using coverage, performance and robustness metrics, and suggest threat intelligence, cybersecurity and online investigation applications.
