Ex-Tesla AI Head: AI Agent Finds GPT-2 Tuning Details in One Night That Humans Missed for Months

For AI researchers, manually tuning model parameters often takes months yet may still miss critical details. Andrej Karpathy, former head of AI and Autopilot at Tesla, recently shared his experience.

He spent months manually adjusting GPT-2 training configurations, but an autonomous AI agent took just one night to find the fine-tuning parameters he'd overlooked—these parameters interact in ways humans easily miss but systematic searches can clearly identify.

Karpathy believes researchers should step back from processes in areas with objective metrics, as they often overtrust their intuition. He noted that major AI labs are systematically automating their work (aligning with their public goals), though such progress may not transfer to hard-to-quantify fields.