Check dimensions before using Normal equations
Write the size and role of every vector, matrix, space, and transformation. An expression can look algebraically familiar while being dimensionally undefined. Rebuild missing background in the linear algebra subject guide, then use linear algebra practice to retrieve the relationship without being told its name.
Follow one object through Normal equations
Track both the coordinates and the geometric or structural meaning:
Carry out the transformation in a valid order
- Name the unknown, and the unit the answer has to come out in.
- Match the symbols to your values. $\widehat{\mathbf x}$ is the least-squares solution.
- Check the conditions before substituting. A unique solution is guaranteed when columns of $A$ are linearly independent; QR is often numerically safer.
- Substitute, keep exact values to the last line, then test the sign, size, and unit against a rough estimate - The check that catches most linear algebra slips.
An order or dimension error to avoid
What goes wrong: Students see Aᵀ on both sides and cancel it, dropping back to Ax = b.
Why it fails: You can only cancel a factor that has an inverse, and Aᵀ is usually rectangular with no inverse at all. Cancelling it asks for an exact solution, which is the thing that does not exist.
Repair: Multiply out both sides and solve the smaller square system instead. Fitting a line through (1, 1), (2, 2), and (3, 2) gives AᵀA = [[3, 6], [6, 14]] and Aᵀb = (5, 11), whose solution is intercept 0.667 and slope 0.5.
Multiply back or test a simple vector
After using Normal equations, repeat the calculation with units attached and test the result against a boundary case or a quick numerical estimate.
When the transformation statement holds
If two columns of A are multiples of each other, AᵀA is singular and there is no unique answer, only a whole family of equally good fits. Even with independent columns the step of forming AᵀA squares the conditioning, so near-duplicate columns lose accuracy fast and a QR factorization is the safer route.
Questions about Normal equations
Why can I not just solve Ax = b directly?
Because with more data points than unknowns there is usually no line passing through every point. The system is inconsistent, so you switch the goal from exact to closest.
Should the residual come out as zero?
Only in the lucky case where the data really does fit exactly. Normally the residual is nonzero, and what makes the answer the best one is that this leftover is perpendicular to every column of A.
What does AᵀA being singular tell me about my data?
That your predictors carry duplicate information, such as one column being a constant multiple of another or a sum of two others. Drop or combine the redundant column and the fit becomes unique.
How big is the system I end up solving?
AᵀA is square with one row and column per unknown, no matter how many data points you have. Fitting a straight line to a million points still leaves you with just a 2 by 2 system.
Change one input, predict whether the output should rise, fall, or stay fixed, and only then recalculate.