Course Content

Eigenvalues, Eigenvectors, and Applications

Two coffee brands share a market. Every month, 90% of brand A's customers stay and 10% defect to B; 80% of B's customers stay and 20% come back to A. Written as a matrix that takes this month's market share to next month's:

M = \begin{bmatrix} 0.9 & 0.2 \\ 0.1 & 0.8 \end{bmatrix}

Start with everyone on brand A, so the share vector is $[1, 0]$ , and run it forward. Month one: $[0.9,\ 0.1]$ . Month two: $[0.83,\ 0.17]$ . Month three: $[0.781,\ 0.219]$ . Month four: $[0.7467,\ 0.2533]$ . Keep going and it settles at $[0.667,\ 0.333]$ and stops moving.

Now start from the opposite extreme, everyone on brand B, $[0, 1]$ . Month one: $[0.2,\ 0.8]$ . Month two: $[0.34,\ 0.66]$ . Carry on and it converges to $[0.667,\ 0.333]$ — the identical answer. Try any starting split you like. Same destination. The market has a two-thirds/one-third equilibrium baked into it, and the starting point is irrelevant.

Check what happens once you are there: $0.9(0.667) + 0.2(0.333) = 0.6 + 0.0667 = 0.667$ . The transformation moved that vector to itself. Almost every vector gets shifted and turned by $M$ , but this one direction is fixed. Find those special directions for a matrix and you have found the skeleton the whole transformation hangs on. They are called eigenvectors, and they are the closest thing linear algebra has to a superpower.

The steady state is the eigenvector for eigenvalue 1 — the one direction the transition matrix leaves alone.

The definition, and what it is really saying

A matrix normally does two things to a vector: turns it and changes its length. An eigenvector is a vector the matrix cannot turn. It comes out pointing exactly the way it went in, only scaled. The scale factor is its eigenvalue.

A\mathbf{v} = \lambda\mathbf{v}, \qquad \mathbf{v} \neq \mathbf{0}

The left side is a full matrix-vector multiply, all the mixing and combining. The right side is one number times the original vector. Saying they are equal is saying the matrix acts like simple multiplication along that particular direction. The condition $\mathbf{v} \neq \mathbf{0}$ is not fussiness: the zero vector satisfies the equation for every $\lambda$ and tells you nothing, so it is excluded by definition.

Read the eigenvalue as a verdict on that direction:

$\lambda > 1$ : vectors along it get stretched, and repeated application blows them up.
$\lambda = 1$ : fixed. Our coffee equilibrium.
$0 < \lambda < 1$ : shrunk, and repeated application drives it to zero.
$\lambda < 0$ : flipped to point the other way, and scaled by $|\lambda|$ .
$\lambda = 0$ : that direction is crushed flat — the matrix is singular.

Eigenvectors come in families. If $\mathbf{v}$ is an eigenvector then so is $3\mathbf{v}$ and $-\mathbf{v}$ , with the same eigenvalue, because only the direction matters. Software returns the unit-length member of each family so the answer is well defined.

An eigenvector is a direction the transformation respects; the eigenvalue is what it does to things pointing that way.

Finding them: the characteristic equation

Rearranging $A\mathbf{v} = \lambda\mathbf{v}$ gives $A\mathbf{v} - \lambda\mathbf{v} = \mathbf{0}$ , and factoring out $\mathbf{v}$ — carefully, inserting the identity $I$ so the subtraction is matrix-shaped — gives

(A - \lambda I)\mathbf{v} = \mathbf{0}

We need a non-zero $\mathbf{v}$ that the matrix $(A - \lambda I)$ sends to zero. A matrix can only flatten a non-zero vector to nothing if it collapses space, and a matrix collapses space exactly when its determinant is zero. So:

\det(A - \lambda I) = 0

That is the characteristic equation. In plain terms: find the values of $\lambda$ that make $A - \lambda I$ singular, because those are the only values with a direction to spare. Work it on

A = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix}

Subtract $\lambda$ from the diagonal and take the determinant:

\det\begin{bmatrix} 4-\lambda & 1 \\ 2 & 3-\lambda \end{bmatrix} = (4-\lambda)(3-\lambda) - (1)(2)

Expand: $12 - 4\lambda - 3\lambda + \lambda^2 - 2 = \lambda^2 - 7\lambda + 10$ . Set it to zero and factor: $(\lambda - 5)(\lambda - 2) = 0$ , so $\lambda_1 = 5$ and $\lambda_2 = 2$ .

Getting the eigenvector for each eigenvalue

Substitute $\lambda = 5$ back in:

(A - 5I) = \begin{bmatrix} -1 & 1 \\ 2 & -2 \end{bmatrix}

The system is $-v_1 + v_2 = 0$ and $2v_1 - 2v_2 = 0$ — the second row is $-2$ times the first, which is expected, because if the rows were independent the only solution would be zero. So the single real constraint is $v_1 = v_2$ , and any multiple of $\mathbf{v}_1 = [1, 1]$ works. Confirm: $A[1,1] = [4+1,\ 2+3] = [5, 5] = 5[1,1]$ .

Now $\lambda = 2$ :

(A - 2I) = \begin{bmatrix} 2 & 1 \\ 2 & 1 \end{bmatrix}

Both rows say $2v_1 + v_2 = 0$ , so $v_2 = -2v_1$ and $\mathbf{v}_2 = [1, -2]$ . Confirm: $A[1,-2] = [4-2,\ 2-6] = [2, -4] = 2[1,-2]$ . Correct.

Two quick sanity checks, both free. The trace — the sum of the diagonal — equals the sum of the eigenvalues: $4 + 3 = 7 = 5 + 2$ . The determinant equals their product: $(4)(3) - (1)(2) = 10 = 5 \times 2$ . If either fails, you have made an arithmetic slip.

Diagonalisation, and why matrix powers become free

Put the eigenvectors in the columns of a matrix $P$ and the eigenvalues down the diagonal of $D$ . Then

A = PDP^{-1}

Read right to left, this is a three-step story. $P^{-1}$ rewrites any vector in terms of the eigenvector directions. $D$ scales each of those directions by its own eigenvalue — the easiest possible operation. $P$ translates back to ordinary coordinates. The tangled matrix $A$ turns out to be plain stretching, viewed in the wrong coordinate system.

The payoff is powers. Computing $A^{20}$ directly means nineteen matrix multiplications. But

A^k = PD^kP^{-1}

because every inner $P^{-1}P$ collapses to the identity: $A^2 = PDP^{-1}PDP^{-1} = PD^2P^{-1}$ , and so on. Raising a diagonal matrix to a power just means raising each diagonal entry to that power. For our $A$ , $D^{20} = \mathrm{diag}(5^{20}, 2^{20})$ , done in two scalar operations. This is exactly why the coffee market converged: run the transition matrix $k$ times and each eigen-direction is multiplied by $\lambda^k$ . The $\lambda = 1$ direction holds steady while everything else decays, so after enough months only the steady state survives.

When it works, and when it does not

Diagonalisation requires $n$ independent eigenvectors for an $n \times n$ matrix. Some matrices do not have them — a pure shear like $\begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$ has $\lambda = 1$ twice but only one eigenvector direction, and cannot be diagonalised. Others have complex eigenvalues, which is a rotation showing up in the algebra.

Symmetric matrices are the happy exception, and they are the ones that matter most in machine learning because covariance matrices, kernel matrices and Hessians are all symmetric. The spectral theorem guarantees that a real symmetric matrix always has a full set of real eigenvalues and eigenvectors that are mutually perpendicular. So $P$ can be built from orthonormal columns, giving $P^{-1} = P^T$ and

A = Q\Lambda Q^T

with no inverse to compute at all. Perpendicular eigenvectors are what make principal components independent of one another rather than partly redundant.

If the matrix is...	Then its eigenvalues...	Practical meaning
Symmetric ( $A = A^T$ )	Are all real, with orthogonal eigenvectors	Safe to decompose; use `eigh`
Positive definite	Are all $> 0$	A bowl-shaped surface with one minimum
Positive semi-definite	Are all $\geq 0$	Valid covariance; any zero eigenvalue is a flat direction
Singular	Include at least one zero	Rank deficient; features are redundant
Triangular	Are the diagonal entries	Read them off directly

Computing them in practice

Python

1import numpy as np23A = np.array([[4.0, 1.0],4              [2.0, 3.0]])56vals, vecs = np.linalg.eig(A)      # general square matrices7# vals  -> [5.+0.j, 2.+0.j]  (NumPy 2.5+ always returns complex here)8# vecs  -> columns, unit length: vecs[:, 0] is the eigenvector for vals[0]910C = np.array([[5.0, 2.0],11              [2.0, 2.0]])         # symmetric: a covariance matrix12vals, vecs = np.linalg.eigh(C)     # ascending order, guaranteed real

Three things trip people up here. Eigenvectors are columns of the returned array, not rows, so pairing vals[i] with vecs[i] is wrong and vecs[:, i] is right. eig can return complex numbers even for a real matrix — recent NumPy releases give a complex array every time, with +0.j on real eigenvalues — and can return them in any order. Use eigh whenever the matrix is symmetric: it is faster, it exploits the structure, it guarantees real output, and it sorts ascending — so the last column is the dominant direction. Finally, a sign flip in an eigenvector is not a bug; $\mathbf{v}$ and $-\mathbf{v}$ are equally valid answers.

Principal component analysis is just this

PCA is the flagship application. Centre your data, form the covariance matrix $C = \frac{1}{n-1}X^TX$ , and decompose it. The eigenvectors are the directions of greatest variance in your data — the principal components — and each eigenvalue is the amount of variance along its direction.

Take a two-feature dataset whose covariance came out as

C = \begin{bmatrix} 5 & 2 \\ 2 & 2 \end{bmatrix}

Characteristic equation: $(5-\lambda)(2-\lambda) - 4 = \lambda^2 - 7\lambda + 6 = 0$ , giving $\lambda = 6$ and $\lambda = 1$ . For $\lambda = 6$ , $(C - 6I) = \begin{bmatrix} -1 & 2 \\ 2 & -4 \end{bmatrix}$ gives $v_1 = 2v_2$ , so the first component points along $[2, 1]$ . For $\lambda = 1$ the direction is $[1, -2]$ — perpendicular, as the spectral theorem promised, since $2(1) + 1(-2) = 0$ .

The variance explained ratio is each eigenvalue over their total:

\frac{6}{6+1} = 85.7\%, \qquad \frac{1}{6+1} = 14.3\%

So a single number per sample — the projection onto $[2,1]$ — retains 85.7% of everything the two original features told you. That is the whole logic of dimensionality reduction: sort the eigenvalues, keep enough of the top ones to reach the variance you need, discard the rest. With 200 features you might find 20 components carrying 95% of the variance, and the discarded 180 directions were mostly noise.

Eigenvalues rank directions by how much they matter. Keeping the big ones and dropping the small ones is compression with a guarantee attached.

The same idea, wearing four other hats

PageRank. Build a matrix where entry $(i,j)$ is the probability a random surfer moves from page $j$ to page $i$ . Columns sum to one, exactly like the coffee market. The dominant eigenvector — the one with $\lambda = 1$ — is the long-run fraction of time spent on each page, which is the ranking. Google computes it by power iteration: multiply a starting vector by the matrix over and over. Every other component shrinks by $|\lambda| < 1$ each round, so the dominant direction takes over. That is precisely what you watched happen with two coffee brands, at the scale of billions of pages.

Spectral clustering. Turn your data into a graph where similar points are connected, build its Laplacian matrix, and look at the eigenvectors with the smallest eigenvalues. Those coordinates pull apart clusters that k-means cannot separate, because they respond to connectivity rather than to straight-line distance — two interleaved crescent shapes, for instance.

Optimisation. At a point where the gradient is zero, the Hessian — the matrix of second derivatives — tells you what kind of point it is, and its eigenvalues give the verdict.

Hessian eigenvalues	Critical point	What training does there
All positive	Local minimum	Curves upward everywhere; you have arrived
All negative	Local maximum	Curves down everywhere; gradient ascent's target
Mixed signs	Saddle point	Down in some directions, up in others — gradients go near zero and progress stalls
Some zero	Flat direction	Inconclusive; the loss is unchanged along it

Saddle points, not local minima, are what actually slow down training in high dimensions: with a thousand parameters, needing all thousand eigenvalues to share a sign is vastly less likely than getting a mixture. The ratio of largest to smallest eigenvalue also sets how badly the loss surface is stretched, which is why plain gradient descent zigzags down long narrow valleys and why adaptive optimisers help.

Deep networks. A signal passing through many layers is repeatedly multiplied by weight matrices, so it is repeatedly scaled by their eigenvalues. If the dominant magnitude sits above 1, then $\lambda^k$ explodes across $k$ layers; below 1, it vanishes. That is the exploding and vanishing gradient problem stated exactly. Orthogonal initialisation, which forces eigenvalue magnitudes to 1, and gradient clipping, which caps the damage, are both direct responses to this arithmetic.

Using this on a real problem

When you next reach for PCA, do not stop at the transformed data. Print the eigenvalues. Their pattern is a diagnosis of your feature set: a sharp drop after a handful of components means your 200 columns were secretly describing about six things, and a nearly flat spectrum means every feature carries independent information and reduction will cost you. A near-zero eigenvalue is a hard signal that two columns are duplicates — the same unit measured twice, or a one-hot encoding where you forgot to drop a category — and it will make any matrix inverse downstream unstable.

Three habits are worth forming. Use eigh, never eig, on anything symmetric, and treat a complex eigenvalue from a matrix you believed was symmetric as evidence you built it wrong. Standardise features before PCA, because eigenvalues chase raw variance and a column measured in millions will monopolise the first component for no better reason than its units. And when a model trains but will not converge, remember that the shape of the loss surface is an eigenvalue question before it is a hyperparameter question.

The pattern under all of it is the one the coffee market showed in four lines of arithmetic: find the directions a transformation leaves alone, and the transformation stops being complicated.