Human-Like Neural Nets by Catapulting (gwern.net) AI

Gwern.net proposes a speculative “catapulted” training paradigm for overparameterized neural nets that uses very high learning rates and regularization on small, filtered datasets to jump into a basin of human-like generalization, with claims of improved robustness to adversarial examples and better alignment prospects. The article frames this as a bias–variance tradeoff idea—LLMs minimizing variance while human brains may minimize bias—and discusses related anomalies such as sample inefficiency, why active learning/embodiment don’t fully explain human learning, and why current architectures or learning rules haven’t yielded clear “magic” from biology.