For LLM-driven optimization, complex meta-heuristics like simulated annealing are unnecessary; simple greedy hill climbing is a superior default.
March 31, 2026
Original Paper
Greedy Is a Strong Default: Agents as Iterative Optimizers
arXiv · 2603.27415
The Takeaway
The authors show that the 'learned prior' of LLMs is so strong that classical acceptance-rule sophistication adds no value and consumes 2-3x more compute. This simplifies the design of agentic optimization loops for hyperparameters, fine-tuning, and discrete search spaces.
From the abstract
Classical optimization algorithms--hill climbing, simulated annealing, population-based methods--generate candidate solutions via random perturbations. We replace the random proposal generator with an LLM agent that reasons about evaluation diagnostics to propose informed candidates, and ask: does the classical optimization machinery still help when the proposer is no longer random? We evaluate on four tasks spanning discrete, mixed, and continuous search spaces (all replicated across 3 independ