Build a simple topic modeling pipeline (BERTopic-style).

What you need to know

Topic modelling answers "what are these 50,000 documents about?" without predefined labels.

The classic method, LDA, models each document as a mixture of word distributions. It works on long documents but struggles on short text — tweets, tickets, reviews — because a 12-word message has too few words to estimate a mixture from.

BERTopic's approach separates the two jobs:

1. Embed — each document becomes a vector; meaning, not word overlap, decides similarity.
2. Reduce — UMAP to about 5 dimensions so density clustering works.
3. Cluster — HDBSCAN finds topic groups and leaves outliers out.
4. Describe — c-TF-IDF picks each cluster's distinguishing words.
5. Name (optional) — an LLM writes a 2–4 word label from keywords and examples.

Class-based TF-IDF (c-TF-IDF). Ordinary TF-IDF scores a word in a document. c-TF-IDF concatenates each cluster into one "class document" and scores words per class:

score(word, cluster) = tf(word, cluster) × log(1 + A / f(word))tf(word, cluster) = count of the word in the cluster / total words in the clusterA                 = average number of words per clusterf(word)           = count of the word across all clusters

A word common in one cluster and rare elsewhere gets a high score; a word common everywhere ("order", "please") gets a low one.

1import re2from collections import Counter3from collections.abc import Callable4import numpy as np56STOPWORDS = {"the", "and", "for", "that", "this", "with", "you", "your", "are", "was",7             "have", "has", "not", "but", "can", "our", "from", "they", "still", "yet"}89def tokenize(text: str) -> list[str]:10    return [w for w in re.findall(r"[a-z]{3,}", text.lower()) if w not in STOPWORDS]1112def c_tf_idf(clusters: dict[int, list[str]], top_n: int = 5) -> dict[int, list[str]]:13    """clusters: {label: [texts]} -> {label: most distinguishing terms}."""14    counts = {c: Counter(w for t in texts for w in tokenize(t)) for c, texts in clusters.items()}15    totals = {c: sum(cnt.values()) for c, cnt in counts.items()}16    avg_words = sum(totals.values()) / len(totals) if totals else 0.017    corpus = sum(counts.values(), Counter())                 # word counts across all clusters18    topics = {}19    for c, cnt in counts.items():20        scored = [(w, (n / totals[c]) * np.log(1 + avg_words / corpus[w])) for w, n in cnt.items()]21        scored.sort(key=lambda pair: (-pair[1], pair[0]))    # score desc, then alphabetical22        topics[c] = [w for w, _ in scored[:top_n]]23    return topics2425def topic_model(texts: list[str], embed_fn: Callable, cluster_fn: Callable,26                name_fn: Callable[[str], str] | None = None) -> list[dict]:27    """cluster_fn(texts, embed_fn) -> {"clusters": {label: [texts]}, ...} (see the HDBSCAN question)."""28    clusters = cluster_fn(texts, embed_fn)["clusters"]29    topics = []30    for label, terms in c_tf_idf(clusters).items():31        name = ", ".join(terms[:3])32        if name_fn:33            examples = "\n".join(f"- {t}" for t in clusters[label][:5])34            name = name_fn(f"Keywords: {', '.join(terms)}\nExamples:\n{examples}\n"35                           "Give a 2-4 word topic name. Reply with the name only.").strip()36        topics.append({"id": label, "name": name, "keywords": terms, "size": len(clusters[label])})37    return sorted(topics, key=lambda t: -t["size"])

The tricky parts:

• sum(counts.values(), Counter()) adds all clusters' counters into one corpus-wide counter; Counter supports +.
• Sorting by (-score, word) breaks ties alphabetically, so results are reproducible run to run.
• Normalising tf by cluster size stops a big cluster's words from dominating just because the cluster has more text.
• cluster_fn is injected, so the same pipeline runs with cluster_texts from the HDBSCAN question (with UMAP as its reduce_fn) or with a stub in tests.

Complexity: embedding dominates. c-TF-IDF is O(total tokens) to count plus O(V log V) per cluster to sort V distinct words. Naming adds one LLM call per topic — tens of calls, not thousands. Memory O(vocabulary × clusters).

A real-life example

c-TF-IDF on three hand-made clusters of delivery-app complaints:

1clusters = {0: ["Refund not received for my order", "Refund still pending after cancelling",2                "When will my refund reach my bank"],3            1: ["Rider was late and the food was cold", "Order arrived late again",4                "Late delivery, cold food"],5            2: ["App crashes when I open my order", "App logs me out after update"]}6for label, terms in c_tf_idf(clusters, top_n=3).items():7    print(label, terms)8# 0 ['refund', 'bank', 'cancelling']9# 1 ['late', 'cold', 'food']10# 2 ['app', 'crashes', 'logs']

Trace for cluster 0: after removing stopwords and words shorter than three letters, it has 12 words; refund appears 3 times in cluster 0 and 3 times in the whole corpus. The average cluster has (12 + 12 + 10) / 3 = 11.33 words.

"order" appears in every cluster, so it scores low even though it is in cluster 0. In cluster 2, "after" and "when" (0.19 each) only just miss the top three — a real pipeline uses a fuller stopword list. That is also why an LLM naming step helps: given the keywords and examples, it can write "App crashes and logouts".

Food-delivery and e-commerce companies run this weekly over free-text feedback, so a new complaint theme shows up as a new topic before it shows up in the ratings.

Follow-up questions to expect

• "The outlier group is huge — is something wrong?" — Often 20–40% outliers is normal for short, noisy text. Read it: it frequently contains emerging topics too small to form a cluster yet. BERTopic can also reassign outliers to their nearest topic afterwards.
• "Topic ids change every run?" — Yes; UMAP and HDBSCAN are not stable identifiers. Match topics across runs by keyword or centroid similarity, never store the raw id as a permanent label.
• "How do you evaluate topics?" — Topic coherence scores, human review of sample documents per topic, and, if labels exist for a subset, agreement with them.