arxiv.org
This arXiv paper proposes an end-to-end scalable architecture for early identification of new Tor sites and daily analysis of their content. It is built on an open source big data stack including Kubernetes, Kafka, Kubeflow and MinIO, discovering onion addresses across several sources, downloading HTML from Tor, deduplicating with MinHash LSH and categorizing with BERTopic modeling. In 93 days it identified 80,049 onion services and characterized 90% of them.