arxiv.org
CRATOR is a proposed dark web crawler described in an arXiv paper. It is designed to extract pages that handle security protocols such as captchas, using seed URL lists, link analysis and scanning to discover content, with user-agent rotation and proxy usage for anonymity. The authors report evaluation using coverage, performance and robustness metrics, and suggest threat intelligence, cybersecurity and online investigation applications.
