Determinants, Matrix Inverse, and Solving Linear Systems

Here is a system of two equations in two unknowns. It looks like every textbook example you have ever seen.

Two equations, two unknowns, so there should be exactly one answer. Try to find it. Double the first equation and you get $4x + 6y = 16$ — which is the second equation, character for character. The second row told you nothing the first row had not already said. So $x = 1, y = 2$ works. So does $x = 4, y = 0$. So does $x = -2.5, y = 13/3$. There are infinitely many solutions and no way to choose between them.

Now change one digit. Make the right-hand side of the second equation 17 instead of 16. The equations now claim that the same quantity $4x + 6y$ equals both 16 and 17. There are zero solutions. The same left-hand side gave us infinitely many answers and then none, depending on a number that never appeared in the coefficients at all.

You could spot this by eye with two equations. With a 500-variable system inside a regression fit, you cannot. You need a single number, computed from the coefficients alone, that tells you in advance whether a unique solution exists. That number is the determinant, and understanding what it actually measures explains not only when systems break but why some models are numerically fragile long before they break outright.

A matrix is a transformation of space

Stop reading $A\mathbf{x}$ as "a table times a list" and start reading it as "a machine that moves points". Feed in a vector, get out a different vector. Feed in every point in the plane and the whole plane gets reshaped in a specific, rigid way: straight lines stay straight, the origin stays put, and evenly spaced points stay evenly spaced. That is what makes it a linear transformation.

• $\begin{bmatrix} 3 & 0 \\ 0 & 2 \end{bmatrix}$ is a scaling: it stretches everything three times horizontally and twice vertically. Reversible — scale back by $1/3$ and $1/2$.
• $\begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}$ is a rotation by 90 degrees anticlockwise. Reversible — rotate the other way.
• $\begin{bmatrix} 1 & 0.5 \\ 0 & 1 \end{bmatrix}$ is a shear: it slides points sideways in proportion to their height, turning a square into a leaning parallelogram. Reversible — shear back.
• $\begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}$ is none of those. It squashes the entire plane onto a single line, and it is not reversible.

The last one is the interesting case. Every input point lands somewhere on the line $y = 2x$. The points $(1,0)$ and $(0,0.5)$ both map to $(1,2)$. Once two different inputs produce the same output, no rule on earth can undo the transformation, because the output does not say which input it came from. Information has been destroyed.

That is the same failure as the two-equations problem at the top, wearing different clothes. Solving $A\mathbf{x} = \mathbf{b}$ means asking "which point lands on $\mathbf{b}$?" If the transformation flattened space onto a line, then points on that line came from infinitely many places, and points off it came from nowhere at all.

The determinant measures how much space is squashed

Take the unit square — corners at $(0,0), (1,0), (0,1), (1,1)$, area exactly 1. Push it through the transformation. It comes out as a parallelogram. The determinant is the area of that parallelogram.

So $\det A = 3$ means every region's area triples. $\det A = 0.5$ means areas halve. A negative determinant means space was flipped over, like turning a page, and the magnitude still gives the area factor. In three dimensions, swap "area" for "volume"; in $n$ dimensions the same idea holds with a word nobody needs.

The determinant is the factor by which a transformation multiplies area. A determinant of zero means area was multiplied by zero — space collapsed, and nothing collapsed can be un-collapsed.

For a $2 \times 2$ matrix the formula is short:

With $A = \begin{bmatrix} 3 & 1 \\ 2 & 4 \end{bmatrix}$: $\det A = (3)(4) - (1)(2) = 12 - 2 = 10$. Areas grow tenfold. Now the coefficient matrix from the opening system, $\begin{bmatrix} 2 & 3 \\ 4 & 6 \end{bmatrix}$: $(2)(6) - (3)(4) = 12 - 12 = 0$. There it is. The determinant knew the system was degenerate without anyone attempting to solve it.

Three by three, by cofactor expansion

For larger matrices you expand along a row: take each entry, multiply it by the determinant of the smaller matrix left when you delete that entry's row and column, and alternate the signs $+, -, +$.

Expanding along the top row:

Each small determinant is $ad - bc$. First: $(-2)(7) - (5)(8) = -14 - 40 = -54$. Second: $(4)(7) - (5)(2) = 28 - 10 = 18$. Third: $(4)(8) - (-2)(2) = 32 + 4 = 36$. Putting it together:

Volumes are multiplied by 306 and space is flipped inside out. Note the cost, though: a $3\times3$ needs three $2\times2$s, a $4\times4$ needs four $3\times3$s, and this grows like $n!$. Nobody computes large determinants this way; software reduces the matrix to triangular form first, where the determinant is just the product of the diagonal.

Properties worth memorising

Singular is simply the technical word for "flattens space, cannot be undone". When a library raises LinAlgError: Singular matrix, this is what it found: two of your features are exact linear combinations of each other, or you have fewer independent samples than parameters, and the maths has nowhere to go.

The inverse: the transformation that undoes the transformation

$A^{-1}$ is the matrix that puts every point back where it started:

where $I$ is the identity — the do-nothing transformation with 1s down the diagonal. An inverse exists exactly when $\det A \neq 0$, which is the same statement as "no information was destroyed", which is the same statement as "$A\mathbf{x} = \mathbf{b}$ has one unique solution for every $\mathbf{b}$". Three phrasings, one fact.

For $2 \times 2$ there is a closed form: swap the diagonal entries, negate the off-diagonal ones, divide by the determinant.

Take $A = \begin{bmatrix} 4 & 7 \\ 2 & 6 \end{bmatrix}$. The determinant is $(4)(6) - (7)(2) = 24 - 14 = 10$, so

Verify by multiplying. Top-left: $4(0.6) + 7(-0.2) = 2.4 - 1.4 = 1$. Top-right: $4(-0.7) + 7(0.4) = -2.8 + 2.8 = 0$. Bottom-left: $2(0.6) + 6(-0.2) = 1.2 - 1.2 = 0$. Bottom-right: $2(-0.7) + 6(0.4) = -1.4 + 2.4 = 1$. That is $I$. Notice where the determinant sits: in the denominator. As $\det A$ approaches zero the entries of the inverse blow up, and that is where numerical trouble begins.

Beyond $2 \times 2$ there is no memorable formula. The practical method is Gauss-Jordan elimination: write $A$ and $I$ side by side, then use row operations — swap two rows, scale a row, add a multiple of one row to another — to grind the left block into $I$. Whatever the right block has become is $A^{-1}$, because you applied the same undo-sequence to both.

Solving $A\mathbf{x} = \mathbf{b}$ the right way

The algebra says: multiply both sides by $A^{-1}$ and get $\mathbf{x} = A^{-1}\mathbf{b}$. The algebra is correct and the code is wrong.

1import numpy as np23x = np.linalg.inv(A) @ b     # correct maths, bad numerics4x = np.linalg.solve(A, b)    # what you should actually write

solve never builds the inverse. It performs LU decomposition — factoring $A$ into a lower-triangular $L$ times an upper-triangular $U$ — and then solves two triangular systems by simple substitution. The difference is not cosmetic:

Seeing $A^{-1}$ in a formula is a licence to solve a system, not an instruction to compute an inverse.

Where this turns up in machine learning

Fitting a linear regression by least squares has an exact closed-form answer, the normal equation:

In words: $X$ is your data with samples in rows and features in columns, $\mathbf{y}$ the targets, and $\mathbf{w}$ the weights that minimise total squared error. $X^TX$ is a small $d \times d$ matrix summarising how features co-vary; inverting it undoes that overlap so each feature gets credit only for what it uniquely explains. This is the formula every textbook prints and the one you should never type literally. If two features are perfectly correlated — height in centimetres and height in inches — then $X^TX$ is singular, the inverse does not exist, and the fit has no unique answer. That is exactly the doubled-equation problem from the opening, arriving through a data-cleaning mistake. Use np.linalg.lstsq, or add a ridge penalty, which adds $\lambda I$ to $X^TX$ and nudges the determinant away from zero.

The other place is covariance matrices. Centre your data, compute $C = \frac{1}{n-1}X^TX$, and entry $C_{ij}$ says how features $i$ and $j$ vary together. Principal component analysis works entirely on this matrix. A zero determinant here means some feature is a perfect combination of others and one direction in your data carries no variance whatsoever — real information, if you know to look for it.

Conditioning: the failure that is worse than failing

Outright singularity at least announces itself. The dangerous case is nearly singular, where everything runs and the answers are rubbish.

Solve $A\mathbf{x} = \begin{bmatrix} 2 \\ 2.001 \end{bmatrix}$. Subtracting the first equation from the second gives $0.001y = 0.001$, so $y = 1$ and $x = 1$.

Now nudge the second entry of $\mathbf{b}$ from $2.001$ to $2.002$ — a change of 0.05%, smaller than any real measurement error. Subtracting gives $0.001y = 0.002$, so $y = 2$ and $x = 0$. The solution moved from $(1,1)$ to $(0,2)$. A rounding-level wobble in the input produced a 100% change in the output.

The quantity that predicts this is the condition number:

where $\sigma$ are the singular values — the stretch factors of the transformation along its most and least stretched directions. So $\kappa$ asks: how lopsided is this map? For our matrix, $\sigma_{\max} \approx 2.0005$ and $\sigma_{\min} \approx 0.0005$, giving $\kappa \approx 4002$. The rule of thumb is that you lose about $\log_{10}\kappa$ digits of accuracy, so roughly 3.6 digits gone, from the 16 that double precision gives you.

Feature scaling is the cheapest fix and the most neglected. If one column is measured in millions and another in hundredths, $\kappa$ is enormous before the model has seen a single data point. Standardising every column to zero mean and unit variance often drops the condition number by several orders of magnitude and costs one line of code.

What to do when you actually build this

Everything above compresses into a short checklist for numerical code that has to work on real data.

Compute np.linalg.cond(A) before you trust any linear solve, and log it. It is one cheap call and it tells you whether the answer you are about to print is meaningful to sixteen digits or to three. Never write np.linalg.inv, because it is slower, less accurate, and almost always a sign that you have transcribed a formula instead of thinking about it — reach for solve, lstsq, or a factorisation. Standardise your features, both because models train better and because it fixes conditioning at the root. And when a solver reports a singular matrix, do not reach for a pseudo-inverse to make the message go away: go and look for the duplicated column, the constant feature, or the one-hot encoding where you forgot to drop a category. The determinant is telling you something true about your data.

The number $ad - bc$ that looked like an arbitrary bit of arithmetic is, in the end, a smoke alarm for your whole pipeline. Learn to hear it.