I.8Singular Values and Singular Vectors in the SVD

The one idea

Eigenvalues are wonderful and they have a fatal limitation: they only exist for square matrices, and even then a matrix may not have a full set of independent eigenvectors. Data matrices are rectangular. You need something better.

The singular value decomposition works for every matrix:

A = UΣVᵀ

with U and V orthogonal and Σ diagonal with non-negative entries σ₁ ≥ σ₂ ≥ ⋯ ≥ 0.

Geometrically A does exactly three things, in order: rotate (Vᵀ), stretch each axis independently (Σ), rotate again (U). That is all any linear map ever does. A sphere goes to an ellipsoid, always. The singular values are the lengths of its semi-axes.

In the rank-one form from I.2:

A = σ₁u₁v₁ᵀ + σ₂u₂v₂ᵀ + ⋯ + σᵣuᵣvᵣᵀ

with r = rank(A) terms, ordered from most to least important.

Where the pieces come from

The defining relation is Avᵢ = σᵢuᵢ. The vᵢ are an orthonormal basis of the input space; A sends each to a scaled member of an orthonormal basis of the output space. Finding two bases that make A diagonal between them is the trick that eigenvectors, using one basis for both, cannot always pull off.

Concretely:

• V and the σᵢ² come from the eigenvectors and eigenvalues of AᵀA
• U and the same σᵢ² come from AAᵀ
• both are symmetric positive semidefinite, so by I.7 they have real non-negative eigenvalues and orthonormal eigenvectors, which is exactly why the SVD always exists

That last point is the payoff for Part I’s ordering. I.7 was not a detour; it is what guarantees this section works.

Why it matters later

Almost everything downstream is the SVD wearing a different hat:

• I.9: the top k terms give the best rank-k approximation (PCA).
• II.2: least squares and the pseudoinverse are the SVD with the zero singular values handled carefully.
• II.3, III.3: how fast σᵢ decays decides whether a matrix compresses well. Fast decay is why images and covariance matrices are so compressible.
• VI, VII: the condition number σ₁/σᵣ governs whether optimization on a problem is well-behaved or crawls.

What to actually do

1. Take A = [[3, 0], [4, 5]]. Form AᵀA, find its eigenvalues, take square roots. Those are σ₁, σ₂. Do it by hand once: the arithmetic is small and the process is what you are learning.
2. In numpy: U, s, Vt = np.linalg.svd(A). Check U @ np.diag(s) @ Vt returns A. Confirm U.T @ U is the identity.
3. Rebuild A from s[0] * np.outer(U[:,0], Vt[0,:]) alone. Compare to the original. How much of the matrix did one rank-one layer already capture?
4. Load any grayscale image as a matrix and repeat step 3 with the top 5, 20, and 50 layers. This makes low-rank approximation something you have seen rather than something you have been told.

Check yourself

• Why does every matrix have an SVD when not every matrix has an eigendecomposition?
• What do the singular values of an orthogonal matrix look like, and why?
• How do you read rank(A) off Σ?
• If you multiply A by 3, what happens to U, Σ, and V individually?

Common sticking points

Confusing singular values with eigenvalues. For a symmetric positive definite matrix they coincide. In general they do not. Singular values are always real and non-negative, eigenvalues can be negative or complex. σᵢ are the square roots of the eigenvalues of AᵀA, which is a different matrix from A.

Sign and ordering ambiguity. You can flip the sign of uᵢ and vᵢ together and nothing changes. Your answer disagreeing with the textbook’s by a sign is usually not an error. Equal singular values leave even more freedom.

Treating Σ as square. For an m×n matrix, Σ is m×n with zeros padding it out. Libraries return the “economy” version by default, which is why np.linalg.svd gives you a 1-D array of singular values instead of a matrix, so you have to build Σ yourself, and getting its shape wrong is the most common reason a reconstruction fails.

---

There is no free section PDF for I.8, but lecture 6 covers it directly and is one of the clearest hours in the course. Watch it before reading, not after.