Implement text clustering using embeddings + HDBSCAN.

What you need to know

Clustering groups items with no labels at all — useful for discovering what customers are complaining about before you have categories.

How HDBSCAN works, in brief. For every point it measures how far you must go to find min_samples neighbours (its "core distance"): small in dense regions, large in sparse ones. It builds a hierarchy of clusters by gradually lowering the density requirement, then keeps the clusters that stay stable over the widest range of densities and are at least min_cluster_size large. Points that never belong to a stable cluster are noise.

Why noise matters for text. A pile of support tickets always contains one-offs: "do you sell gift cards?". K-means puts them into the nearest cluster and blurs its topic; HDBSCAN leaves them out.

Why reduce dimensions first. In 768 or 1,536 dimensions, distances between all points become similar, so "dense region" loses meaning. UMAP to 5–15 dimensions restores contrast (the next question shows the effect).

1from collections.abc import Callable2import numpy as np3from sklearn.cluster import HDBSCAN45def cluster_texts(texts: list[str], embed_fn: Callable, min_cluster_size: int = 5,6                  min_samples: int | None = None, reduce_fn: Callable | None = None) -> dict:7    """Group texts by embedding density; -1 (noise) is returned separately."""8    if len(texts) < min_cluster_size:9        return {"clusters": {}, "noise": list(texts), "noise_ratio": 1.0 if texts else 0.0}10    X = np.asarray(embed_fn(texts), dtype=np.float32)11    X /= np.linalg.norm(X, axis=1, keepdims=True) + 1e-1012    if reduce_fn is not None:13        X = reduce_fn(X)                               # e.g. UMAP to 5 dimensions14    labels = HDBSCAN(min_cluster_size=min_cluster_size, min_samples=min_samples,15                     copy=True).fit_predict(X)         # Euclidean on unit vectors ~ cosine16    clusters: dict[int, list[str]] = {}17    for text, label in zip(texts, labels):18        clusters.setdefault(int(label), []).append(text)19    noise = clusters.pop(-1, [])20    return {"clusters": clusters, "noise": noise, "noise_ratio": round(len(noise) / len(texts), 3)}

The tricky parts:

• Normalise, then use Euclidean distance. For unit vectors, squared Euclidean distance is 2 − 2 × cosine, so neighbours are the same as under cosine, and HDBSCAN's fast Euclidean code paths apply.
• clusters.pop(-1, []) separates the noise so callers cannot mistake it for a cluster called "-1".
• The early return when there are fewer texts than min_cluster_size: no cluster could form anyway.
• copy=True stops scikit-learn from modifying the input array in place (and silences its warning about the default changing).

Complexity: embedding n texts dominates. HDBSCAN builds a neighbour graph and a minimum spanning tree: roughly O(n log n) in low dimensions with tree indexes, approaching O(n²) in high dimensions. Memory O(n·d).

A real-life example

Synthetic data with a known answer: 60 "tickets" around three topic directions (20 each) in 8 dimensions, plus 5 random one-off tickets:

1rng = np.random.default_rng(2)2topics = np.eye(8)[:3]3X = np.vstack([t + rng.normal(0, 0.08, (20, 8)) for t in topics] + [rng.normal(0, 1, (5, 8))])4texts = [f"refund {i}" for i in range(20)] + [f"delivery {i}" for i in range(20)] + \5        [f"login {i}" for i in range(20)] + [f"one-off {i}" for i in range(5)]6vectors = dict(zip(texts, X))7result = cluster_texts(texts, lambda ts: [vectors[t] for t in ts], min_cluster_size=5)8print({k: len(v) for k, v in result["clusters"].items()}, result["noise"], result["noise_ratio"])9# {1: 21, 0: 20, 2: 20} ['one-off 0', 'one-off 1', 'one-off 2', 'one-off 4'] 0.062

Three clusters were found without being told "3", and four of the five random tickets were left out. The fifth random vector happened to point close to one topic — HDBSCAN cannot know it was random, and neither could a person looking only at its position. On real data, you read a sample of each cluster and of the noise to check.

A food-delivery company clustering a week of "other" category complaints finds groups like "rider asked for extra money" that no one had created a category for.

Follow-up questions to expect

• "Everything came out as noise — why?" — Usually no dimensionality reduction on high-dimensional embeddings, or min_cluster_size / min_samples too high. Reduce with UMAP first and lower min_samples.
• "How do you assign new texts later?" — HDBSCAN has no predict for new points in scikit-learn. The standalone hdbscan package offers approximate_predict (fit with prediction_data=True); or assign new texts to the nearest cluster centroid with a distance cut-off.
• "How do you judge cluster quality without labels?" — Read samples from each cluster, check the noise ratio, and use internal scores such as DBCV (built for density clusters) with care.