The Training Example Lie Bracket (pbement.com) AI
The article frames each training example’s gradient update as a vector field over model parameters and uses the Lie bracket of two such fields to quantify how much the final parameters (and predictions) change when two examples are swapped in order. It derives the Lie-bracket term as the leading O(ε²) difference between “x then y” vs “y then x” updates, and then computes these brackets across checkpoints for a small convnet on CelebA. The author reports that Lie-bracket magnitudes track gradient magnitudes closely, and that certain attribute pairs (e.g., Black_Hair vs Brown_Hair) show unusually large logit changes, suggesting possible issues with the assumed independence structure in the loss.