Can LLMs Beat Classical Hyperparameter Optimization Algorithms? (arxiv.org) AI

The arXiv study tests whether LLM-based agents can outperform classical hyperparameter optimization (HPO) methods when tuning a small language model under a fixed compute budget, finding that classical CMA-ES and TPE generally do better, largely due to reducing out-of-memory failures. Allowing LLM agents to edit training code narrows the gap but does not eliminate it, and the authors propose a hybrid method, Centaur, that combines CMA-ES’s internal state with an LLM; Centaur performs best, with an 0.8B LLM sufficient to beat prior classical and pure LLM approaches. The results suggest LLMs are more effective as a complement to classical optimizers than as a replacement.