Nvidia Forms AI Safety Team, Betting on Open-Weight Models

Nvidia is quietly building a new AI safety and cybersecurity engineering team. The team will assess security risks of AI agents before they're deployed and develop tools that use AI to automatically fix software bugs. This move shows Nvidia is taking AI safety more seriously as it bets on open-weight models and AI agents.

Open-weight models share the training parameters that determine how the model works, but training data and source code can stay private. Nvidia said in a job posting that open-weight models, transparency, and broad scientific review are key to keeping the US ahead in AI and strengthening cybersecurity.

In recent months, Nvidia has increasingly supported open-weight AI models, unlike OpenAI and Anthropic which use closed models. CEO Jensen Huang posted on X for the first time last month, sharing an open letter that calls on US policymakers to support open models. He said open models can improve AI safety and boost cybersecurity.

As AI moves from chatbots to agents that can read sensitive company data and take real actions, whether businesses trust AI systems will be key to how fast they're adopted.