Disordered systems · machine learning

Spin Glass Analysis of Neural Network Training

Spin Glass Analysis of Neural Network Training
fig. Spin Glass Analysis of Neural Network Training

The notebook · live in your browser, code and all

open full screen ↗

The notebook runs on its own once it loads, so give it about 10 seconds while Python starts up in your browser, and the simulations begin animating. Move the sliders and everything recomputes live. The code is shown alongside the output so you can read exactly how it works; the full editable source is linked below.

Training a neural network is really a walk downhill on a bumpy, high-dimensional surface, and that surface behaves a lot like a spin glass. I train a small network and watch the curvature of the loss landscape through the diagonal of the Hessian. The inverse participation ratio of that curvature spectrum tells me how the minimum is shaped: spread flat across many directions early on, the glassy phase, then concentrating into a few sharp directions as training settles. It is a small, hands-on way into the physics behind the 2024 Nobel.

Code & resources

← back to explorations