A working paper by John Schriner that focuses on how researchers monitor the Dark Web. It defines onion services and Tor and discusses tools for monitoring and securing onion services.
Dizzy is an open-source crawling and analysis system for onion services. It implements techniques to explore, update, check, and classify onion services at scale without overwhelming the Tor network. It was deployed in April 2021 and used to analyze more than 63.3 million crawled onion webpages, focusing on domain operations, web content, cryptocurrency usage, and web graph.
An arXiv paper presenting a large-scale analysis of the Tor network using a crawler called Mimir, which collects and visits linked content. The authors report a dataset of pages from more than 25k sites and examine network topology, replicated content and content types.
An arXiv paper presenting a longitudinal analysis of the dark web cybercrime ecosystem. The authors analyze 25,065 websites using 11,403,638 HTML snapshots collected over six years, applying a topic-modeling framework with domain-specific embeddings, density-based clustering and temporal aggregation. The analysis identifies 55 thematic clusters and measures topic prevalence and lifecycle at the website level.
Study of whether website fingerprinting can distinguish and identify Tor onion-service traffic.
This arXiv paper proposes an end-to-end scalable architecture for early identification of new Tor sites and daily analysis of their content. It is built on an open source big data stack including Kubernetes, Kafka, Kubeflow and MinIO, discovering onion addresses across several sources, downloading HTML from Tor, deduplicating with MinHash LSH and categorizing with BERTopic modeling. In 93 days it identified 80,049 onion services and characterized 90% of them.
Public-health research measuring harmful-content availability and searches on the Tor network.
Community notes
Be the first to share how this project worked for you.
Sign in with a verified account to leave a review or join the discussion.
Sign in to reviewKeep exploring
Dizzy is an open-source crawling and analysis system for onion services. It implements techniques to explore, update, check, and classify onion services at scale without overwhelming the Tor network. It was deployed in April 2021 and used to analyze more than 63.3 million crawled onion webpages, focusing on domain operations, web content, cryptocurrency usage, and web graph.
An arXiv paper presenting a large-scale analysis of the Tor network using a crawler called Mimir, which collects and visits linked content. The authors report a dataset of pages from more than 25k sites and examine network topology, replicated content and content types.
An arXiv paper presenting a longitudinal analysis of the dark web cybercrime ecosystem. The authors analyze 25,065 websites using 11,403,638 HTML snapshots collected over six years, applying a topic-modeling framework with domain-specific embeddings, density-based clustering and temporal aggregation. The analysis identifies 55 thematic clusters and measures topic prevalence and lifecycle at the website level.
Study of whether website fingerprinting can distinguish and identify Tor onion-service traffic.
This arXiv paper proposes an end-to-end scalable architecture for early identification of new Tor sites and daily analysis of their content. It is built on an open source big data stack including Kubernetes, Kafka, Kubeflow and MinIO, discovering onion addresses across several sources, downloading HTML from Tor, deduplicating with MinHash LSH and categorizing with BERTopic modeling. In 93 days it identified 80,049 onion services and characterized 90% of them.
Public-health research measuring harmful-content availability and searches on the Tor network.
