Name: Linear Algebra and Learning from Data
Author: Gilbert Strang
ISBN: 978-0-692-19638-0

Highlights of Linear Algebra

Column spaces, factorizations, eigenvalues, SVD: the foundation.

Part I progress: 0/11 sections read

Mark section I.1, Multiplication Ax Using Columns of A, as read
Multiplication Ax Using Columns of A
Read the section, opens in a new tab Lecture 1, The Column Space of A Contains All Vectors Ax, opens in a new tab Problem set, opens in a new tab
Guide
Mark section I.2, Matrix-Matrix Multiplication AB, as read
Matrix-Matrix Multiplication AB
Read the section, opens in a new tab Lecture 2, Multiplying and Factoring Matrices, opens in a new tab Problem set, opens in a new tab
Guide
Mark section I.3, The Four Fundamental Subspaces, as read
The Four Fundamental Subspaces
Problem set, opens in a new tab Locate in the contents, opens in a new tab
Mark section I.4, Elimination and A = LU, as read
Elimination and A = LU
Problem set, opens in a new tab Locate in the contents, opens in a new tab
Mark section I.5, Orthogonal Matrices and Subspaces, as read
Orthogonal Matrices and Subspaces
Lecture 3, Orthonormal Columns in Q Give Q'Q = I, opens in a new tab Problem set, opens in a new tab
Mark section I.6, Eigenvalues and Eigenvectors, as read
Eigenvalues and Eigenvectors
Lecture 4, Eigenvalues and Eigenvectors, opens in a new tab Problem set, opens in a new tab
Mark section I.7, Symmetric Positive Definite Matrices, as read
Symmetric Positive Definite Matrices
Lecture 5, Positive Definite and Semidefinite Matrices, opens in a new tab Problem set, opens in a new tab
Mark section I.8, Singular Values and Singular Vectors in the SVD, as read
Singular Values and Singular Vectors in the SVD
Lecture 6, Singular Value Decomposition (SVD), opens in a new tab Problem set, opens in a new tab
Guide
Mark section I.9, Principal Components and the Best Low Rank Matrix, as read
Principal Components and the Best Low Rank Matrix
Lecture 7, Eckart-Young: The Closest Rank k Matrix to A, opens in a new tab Problem set, opens in a new tab
Guide
Mark section I.10, Rayleigh Quotients and Generalized Eigenvalues, as read
Rayleigh Quotients and Generalized Eigenvalues
Problem set, opens in a new tab Locate in the contents, opens in a new tab
Mark section I.11, Norms of Vectors and Functions and Matrices, as read
Norms of Vectors and Functions and Matrices
Lecture 8, Norms of Vectors and Matrices, opens in a new tab Lecture 11, Minimizing the Norm of x Subject to Ax = b, opens in a new tab Problem set, opens in a new tab

Before moving on from Part I

Redo at least one worked example by hand, not just reading it
Write the 4 fundamental subspaces from memory, with their dimensions
State the SVD in your own words: what U, Σ, and V each represent
Do 2-3 problems from the problem set PDF before checking solutions

Computations with Large Matrices

Numerical linear algebra, least squares, randomized methods.

Part II progress: 0/4 sections read

Mark section II.1, Numerical Linear Algebra, as read
Numerical Linear Algebra
Lecture 12, Computing Eigenvalues and Singular Values, opens in a new tab Problem set, opens in a new tab
Mark section II.2, Least Squares: Four Ways, as read
Least Squares: Four Ways
Lecture 9, Four Ways to Solve Least Squares Problems, opens in a new tab Problem set, opens in a new tab
Mark section II.3, Three Bases for the Column Space, as read
Three Bases for the Column Space
Lecture 10, Survey of Difficulties with Ax = b, opens in a new tab Problem set, opens in a new tab
Mark section II.4, Randomized Linear Algebra, as read
Randomized Linear Algebra
Lecture 13, Randomized Matrix Multiplication, opens in a new tab Problem set, opens in a new tab

Before moving on from Part II

List the 4 ways to solve least squares and when each one is preferred
Explain why randomized methods scale to matrices too big to store
Code a least-squares solve in numpy and compare it to the normal equations
Note the stability problems that motivate each method

Low Rank and Compressed Sensing

How matrices change, interlacing eigenvalues, decaying singular values.

Part III progress: 0/5 sections read

Mark section III.1, Changes in A⁻¹ from Changes in A, as read
Changes in A⁻¹ from Changes in A
Lecture 14, Low Rank Changes in A and Its Inverse, opens in a new tab Lecture 15, Matrices A(t) Depending on t, Derivative = dA/dt, opens in a new tab Problem set, opens in a new tab
Mark section III.2, Interlacing Eigenvalues and Low Rank Signals, as read
Interlacing Eigenvalues and Low Rank Signals
Lecture 16, Derivatives of Inverse and Singular Values, opens in a new tab Lecture 18, Counting Parameters in SVD, LU, QR, Saddle Points, opens in a new tab Problem set, opens in a new tab
Mark section III.3, Rapidly Decaying Singular Values, as read
Rapidly Decaying Singular Values
Lecture 17, Rapidly Decreasing Singular Values, opens in a new tab Problem set, opens in a new tab
Mark section III.4, Split Algorithms for ℓ² + ℓ¹, as read
Split Algorithms for ℓ² + ℓ¹
Problem set, opens in a new tab Locate in the contents, opens in a new tab
Mark section III.5, Compressed Sensing and Matrix Completion, as read
Compressed Sensing and Matrix Completion
Problem set, opens in a new tab Locate in the contents, opens in a new tab

Before moving on from Part III

Say in one sentence why low-rank updates matter computationally
Connect rapidly decaying singular values to real data compression
Sketch why interlacing bounds eigenvalues after a rank-one update
Skim Part VII to see where all of this reappears in deep learning

Special Matrices

Circulants, Fourier, graphs, clustering, distance matrices.

Part IV progress: 0/10 sections read

Before moving on from Part IV

Draw a small graph and build its Laplacian matrix by hand
Explain how spectral clustering uses eigenvectors of the Laplacian
Verify on a tiny circulant that its eigenvectors are Fourier modes
Name one real application per matrix type (image, audio, network)

Probability and Statistics

Mean, variance, covariance: the statistical toolkit behind learning.

Part V progress: 0/6 sections read

Mark section V.1, Mean, Variance, and Probability, as read
Mean, Variance, and Probability
Lecture 19, Saddle Points Continued, Maxmin Principle, opens in a new tab Problem set, opens in a new tab
Mark section V.2, Probability Distributions, as read
Probability Distributions
Lecture 20, Definitions and Inequalities, opens in a new tab Problem set, opens in a new tab
Mark section V.3, Moments, Cumulants, and Inequalities of Statistics, as read
Moments, Cumulants, and Inequalities of Statistics
Lecture 20, Definitions and Inequalities, opens in a new tab Problem set, opens in a new tab
Mark section V.4, Covariance Matrices and Joint Probabilities, as read
Covariance Matrices and Joint Probabilities
Problem set, opens in a new tab Locate in the contents, opens in a new tab
Mark section V.5, Multivariate Gaussian and Weighted Least Squares, as read
Multivariate Gaussian and Weighted Least Squares
Problem set, opens in a new tab Locate in the contents, opens in a new tab
Mark section V.6, Markov Chains, as read
Markov Chains
Problem set, opens in a new tab Locate in the contents, opens in a new tab

Before moving on from Part V

Write mean, variance, and covariance definitions side by side
Connect the covariance matrix back to PCA from section I.9
Work one Markov chain example all the way to steady state
Relate weighted least squares back to Part II's four methods

Optimization

Convexity, Lagrange multipliers, gradient descent, duality.

Part VI progress: 0/5 sections read

Mark section VI.1, Minimum Problems: Convexity and Newton's Method, as read
Minimum Problems: Convexity and Newton's Method
Lecture 21, Minimizing a Function Step by Step, opens in a new tab Problem set, opens in a new tab
Mark section VI.2, Lagrange Multipliers = Derivatives of the Cost, as read
Lagrange Multipliers = Derivatives of the Cost
Lecture 24, Linear Programming and Two-Person Games, opens in a new tab Problem set, opens in a new tab
Mark section VI.3, Linear Programming, Game Theory, and Duality, as read
Linear Programming, Game Theory, and Duality
Lecture 24, Linear Programming and Two-Person Games, opens in a new tab Problem set, opens in a new tab
Mark section VI.4, Gradient Descent Toward the Minimum, as read
Gradient Descent Toward the Minimum
Lecture 22, Gradient Descent: Downhill to a Minimum, opens in a new tab Lecture 23, Accelerating Gradient Descent (Use Momentum), opens in a new tab Problem set, opens in a new tab
Mark section VI.5, Stochastic Gradient Descent and ADAM, as read
Stochastic Gradient Descent and ADAM
Lecture 25, Stochastic Gradient Descent, opens in a new tab Problem set, opens in a new tab

Before moving on from Part VI

Derive the gradient descent update from a Taylor expansion sketch
Explain momentum in one or two sentences, in your own words
Compare batch gradient descent against the stochastic version
Code a basic gradient descent loop on a toy quadratic function

Learning from Data

Neural network architecture, convolutions, backpropagation.

Part VII progress: 0/5 sections read

Mark section VII.1, The Construction of Deep Neural Networks, as read
The Construction of Deep Neural Networks
Read the section, opens in a new tab Lecture 26, Structure of Neural Nets for Deep Learning, opens in a new tab Lecture 33, Neural Nets and the Learning Function, opens in a new tab Problem set, opens in a new tab
Guide
Mark section VII.2, Convolutional Neural Nets, as read
Convolutional Neural Nets
Lecture 32, ImageNet is a CNN, The Convolution Rule, opens in a new tab Lecture 36, Alan Edelman and Julia Language, opens in a new tab Problem set, opens in a new tab
Mark section VII.3, Backpropagation and the Chain Rule, as read
Backpropagation and the Chain Rule
Lecture 27, Backpropagation: Find Partial Derivatives, opens in a new tab Problem set, opens in a new tab
Mark section VII.4, Hyperparameters: The Fateful Decisions, as read
Hyperparameters: The Fateful Decisions
Problem set, opens in a new tab Locate in the contents, opens in a new tab
Mark section VII.5, The World of Machine Learning, as read
The World of Machine Learning
Problem set, opens in a new tab Locate in the contents, opens in a new tab

Before moving on from Part VII

Sketch a small net and label every weight, bias, and activation
Trace one forward pass and one backward pass by hand on a tiny net
Explain why convolution is a structured, low-parameter linear map
Write down how Parts I, III, and VI each feed into this chapter

Linear Algebra and Learning from Data

Multiplication Ax Using Columns of A

Matrix-Matrix Multiplication AB

The Four Fundamental Subspaces

Elimination and A = LU

Orthogonal Matrices and Subspaces

Eigenvalues and Eigenvectors

Symmetric Positive Definite Matrices

Singular Values and Singular Vectors in the SVD

Principal Components and the Best Low Rank Matrix

Rayleigh Quotients and Generalized Eigenvalues

Norms of Vectors and Functions and Matrices

Before moving on from Part I

Numerical Linear Algebra

Least Squares: Four Ways

Three Bases for the Column Space

Randomized Linear Algebra

Before moving on from Part II

Changes in A⁻¹ from Changes in A

Interlacing Eigenvalues and Low Rank Signals

Rapidly Decaying Singular Values

Split Algorithms for ℓ² + ℓ¹

Compressed Sensing and Matrix Completion

Before moving on from Part III

Fourier Transforms: Discrete and Continuous

Shift Matrices and Circulant Matrices

The Kronecker Product A ⊗ B

Sine and Cosine Transforms from Kronecker Sums

Toeplitz Matrices and Shift Invariant Filters

Graphs and Laplacians and Kirchhoff's Laws

Clustering by Spectral Methods and k-means

Completing Rank One Matrices

The Orthogonal Procrustes Problem

Distance Matrices

Before moving on from Part IV

Mean, Variance, and Probability

Probability Distributions

Moments, Cumulants, and Inequalities of Statistics

Covariance Matrices and Joint Probabilities

Multivariate Gaussian and Weighted Least Squares

Markov Chains

Before moving on from Part V

Minimum Problems: Convexity and Newton's Method

Lagrange Multipliers = Derivatives of the Cost

Linear Programming, Game Theory, and Duality

Gradient Descent Toward the Minimum

Stochastic Gradient Descent and ADAM

Before moving on from Part VI

The Construction of Deep Neural Networks

Convolutional Neural Nets

Backpropagation and the Chain Rule

Hyperparameters: The Fateful Decisions

The World of Machine Learning

Before moving on from Part VII

Material for the whole book

Attribution