Topics in Machine Learning: Neural Network Training Dynamics

Lecture notes for courses Topics in Machine Learning: Neural Network Training Dynamics.

Lecture Notes

📖 Foundational Concepts

• A Toy Model: Linear Regression

Linear Regression, Gradient Descent, Invariance to Rigid Transformations(Eigenbasis, Curvature)

Convergence Analysis: Coordinatewise Dynamics, Minimum-Cost Subspace, Speed of Convergence(Condition Number), Implicit Regularization

• Normalization, Double Descent

Why Normalize the Features: Normalization, Standardization, Whitening

Double Descent(Interpolation Threshold, Overparameterization)

• Taylor Approximations

Jacobian(JVP/VJP), Hessian(HVP, Rayleigh Quotient), Hessian Spectrum

Example: Weak Symmetry Breaking in Regularized Linear Autoencoders

💡 Understanding Neural Networks

🎛 Game Dynamics and Bilevel Optimization

:mag: Schedule