However, human judgement suggested that mid-sized topic models produced expressive low-dimensional summarizations of the corpus. Few model evaluation metrics suggested mid-sized topic models (25 ≤ K ≤ 75) as being optimal. Rank biased overlap and Kendall’s weighted tau favored small models (K = 5 topics). Stability analysis using topic coherence and the Xie-Beni statistic also favored large models (K = 100 topics). ResultsĬross-validated reconstruction error favored large topic models (K ≥ 100 topics) on our corpus. Depending on context, cross-validation and/or bootstrap stability analysis were used to estimate these metrics on our corpus. We explored the following metrics: reconstruction error, topic coherence, rank biased overlap, Kendall’s weighted tau, partition coefficient, partition entropy and the Xie-Beni statistic. Several topic model quality metrics have been proposed to assess different aspects of model fit. Using a retrospective cohort design, we curated a text corpus containing 382,666 clinical notes collected between through from primary care electronic medical records in Toronto Canada. We synthesize this literature, discuss the advantages and disadvantages of different techniques for topic model validation, and illustrate their usefulness for guiding model selection on a large clinical text corpus. Techniques for topic model validation have been developed across disparate fields. This study reviews several methods for assessing the quality of unsupervised topic models estimated using non-negative matrix factorization.

Topic models are a class of unsupervised machine learning models, which facilitate summarization, browsing and retrieval from large unstructured document collections.

Coherence score umass interpret