Live Coding Interview Prep

Course Content

Live Coding Interview Prep

7 sections · 50 lessons

Apply dimensionality reduction (PCA/UMAP) before clustering.

HDBSCAN on 600 noisy 384-d vectors, 4 true groups75.2%0.06367.3%0.0380%1.0labelled noiseadjusted Randraw 384-dPCA to 50PCA then UMAP 5
In 384 dimensions density means little; UMAP keeps each point's neighbours and gives HDBSCAN clusters it can find.

What you need to know

The curse of dimensionality. In very high dimensions, random noise adds up across hundreds of coordinates, and the distance to a point's nearest neighbour becomes almost the same as the distance to its farthest. A density-based method like HDBSCAN then sees "everything is equally far from everything" and calls most points noise.

Dimensionality reduction maps each vector to far fewer numbers while keeping the structure that matters.

PCA

  • Linear: projects onto the directions of largest variance
  • Fast, deterministic, has transform for new data
  • explained_variance_ratio_ says how much variance you kept
  • Keeps global structure; weak at curved or tangled clusters

UMAP

  • Non-linear: preserves each point's nearest neighbours
  • Slower; results vary with the random seed
  • Excellent at separating clusters
  • Distances between clusters in the output mean little

The usual recipe (the one BERTopic uses): PCA to ~50 dimensions as cheap denoising and speed-up, then UMAP to ~5 for clustering. Use 2 dimensions only for pictures, and never read cluster sizes or gaps off a UMAP plot as if they were real distances.

Python
import numpy as npfrom sklearn.decomposition import PCAdef reduce_pca(X: np.ndarray, n_components: int = 50, seed: int = 42) -> tuple[np.ndarray, float]:    n_components = min(n_components, X.shape[0], X.shape[1])    pca = PCA(n_components=n_components, random_state=seed)    Z = pca.fit_transform(X)    return Z, float(pca.explained_variance_ratio_.sum())def reduce_umap(X: np.ndarray, n_components: int = 5, n_neighbors: int = 15,                seed: int = 42) -> np.ndarray:    import umap                                      # pip install umap-learn    n_neighbors = min(n_neighbors, len(X) - 1)    reducer = umap.UMAP(n_components=n_components, n_neighbors=n_neighbors,                        min_dist=0.0, metric="cosine", random_state=seed)    return reducer.fit_transform(X)def prepare_for_clustering(X: np.ndarray) -> np.ndarray:    """Unit vectors -> PCA 50 (denoise) -> UMAP 5 (cluster-friendly)."""    X = X / (np.linalg.norm(X, axis=1, keepdims=True) + 1e-10)    Z, _ = reduce_pca(X, n_components=50)    return reduce_umap(Z, n_components=5)

The tricky parts:

  • Clamping n_components — PCA cannot return more components than there are samples or features.
  • Clamping n_neighbors — UMAP needs fewer neighbours than points; with 10 documents and the default 15 it fails.
  • min_dist=0.0 lets UMAP place similar points on top of each other, which sharpens density for clustering. For a readable 2-D plot, use something like 0.1–0.5.
  • random_state makes UMAP reproducible, at the cost of running single-threaded.

Complexity: PCA via randomised SVD is about O(n·d·k) for k components. UMAP builds an approximate nearest-neighbour graph (roughly O(n log n)), then optimises the layout, and is the slowest step. Memory O(n·d).

A real-life example

A synthetic test with a known answer: 600 unit vectors in 384 dimensions, 150 around each of 4 topic directions, with heavy per-coordinate noise (standard deviation 0.15), then clustered with HDBSCAN (min_cluster_size=15) three ways:

Python
from sklearn.cluster import HDBSCANfrom sklearn.metrics import adjusted_rand_scorerng = np.random.default_rng(7)topics = rng.normal(size=(4, 384))topics /= np.linalg.norm(topics, axis=1, keepdims=True)y = np.repeat(np.arange(4), 150)X = topics[y] + rng.normal(0, 0.15, (600, 384))X /= np.linalg.norm(X, axis=1, keepdims=True)def score(Z):    labels = HDBSCAN(min_cluster_size=15, copy=True).fit_predict(Z)    return round(float((labels == -1).mean()), 3), round(adjusted_rand_score(y, labels), 3)Z, kept = reduce_pca(X, 50)print("raw 384-d:", score(X))                         # raw 384-d: (0.752, 0.063)print("PCA 50-d:", score(Z), round(kept, 3))           # PCA 50-d: (0.673, 0.038) 0.367print("PCA+UMAP 5-d:", score(reduce_umap(Z, 5)))       # PCA+UMAP 5-d: (0.0, 1.0)
input to HDBSCANshare labelled noiseadjusted Rand index vs truth
raw 384 dimensions75.2%0.063 (almost random)
PCA to 5067.3%0.038
PCA 50 → UMAP 50%1.0 (perfect)

The adjusted Rand index compares a clustering with the true groups: 1 is a perfect match, around 0 is chance. Raw, HDBSCAN called three quarters of the points noise. PCA alone kept only 37% of the variance and did not help. UMAP, which preserves each point's neighbourhood, recovered all four groups exactly. Real embeddings are less extreme, but the direction of the effect is the same.

Product analytics teams clustering tens of thousands of app-store reviews run this PCA-then-UMAP step before HDBSCAN as a matter of course.

Follow-up questions to expect

  • "Why not UMAP straight from 768 dimensions?" — It works; PCA first mainly makes UMAP faster and slightly less noisy, with little loss.
  • "New documents arrive — do you refit?" — Use the fitted reducers' transform for new points so they land in the same space; refit periodically on the full corpus, and accept that cluster ids change when you do.
  • "How do you pick the number of UMAP dimensions?" — 5–15 works for most text; check that cluster quality (on a labelled sample, or by reading clusters) is stable across a couple of values.