TextCL

TextCL is a text preprocessing package intended for cleaning text data before use in natural language processing tasks. It can be an initial step in text analysis and in predictive, classification, or text generation models. Documented preprocessing tasks include splitting texts into sentences, filtering by language, perplexity filtering, duplicate sentence filtering using Jaccard similarity, and unsupervised outlier detection with methods such as TONMF, RPCA, and SVD.

GitHub repo Directory import
0 shouts Added Sep 23, 2026 github.com
github.com/alinapetukhova/textcl
TextCL website preview

Features

  • Duplicate sentences filtering using Jaccard similarity Source: github.com

Links

Tags