Search results

Dark Web Architecture Study

arxiv.org

This arXiv paper proposes an end-to-end scalable architecture for early identification of new Tor sites and daily analysis of their content. It is built on an open source big data stack including Kubernetes, Kafka, Kubeflow and MinIO, discovering onion addresses across several sources, downloading HTML from Tor, deduplicating with MinHash LSH and categorizing with BERTopic modeling. In 93 days it identified 80,049 onion services and characterized 90% of them.

Dizzy

arxiv.org

Dizzy is an open-source crawling and analysis system for onion services. It implements techniques to explore, update, check, and classify onion services at scale without overwhelming the Tor network. It was deployed in April 2021 and used to analyze more than 63.3 million crawled onion webpages, focusing on domain operations, web content, cryptocurrency usage, and web graph.

Recorded Future Insikt Group

recordedfuture.com

Insikt Group is the research unit of Recorded Future. It conducts research on cybercrime, state-sponsored threats and global issues, and supplies analyst-validated data, analytics and cyber and geopolitical intelligence to the Recorded Future Platform. Its resources include intelligence reports, threat briefings and information on threat actors.