Huawei Ascend Achieves Training-Inference Consistency, Qwen3-30B Logdiff Hits Zero

Reinforcement learning has become the mainstream method for training large models, and consistency between training and inference is crucial. Huawei Computing recently announced that it provides a complete set of operators for training-inference consistency based on the Ascend C programming language, achieving a Logdiff of 0 on the Qwen3-30B MoE model. With Ascend-optimized FA operators, it also gains a 20%-60% performance boost in Eager mode.

The inconsistency stems from non-deterministic accumulation order in low-level computations, which worsens with larger models. Ascend addresses this by aligning attention semantics, locking reduce accumulation order, standardizing online Softmax chunking, and aligning precision on critical paths, ensuring training and inference follow the same numerical path without sacrificing performance.

Huawei has open-sourced the related infrastructure and plans to launch a 'Training-Inference Consistency Issue Identification Skill' to help developers debug residual logdiff and identify root causes.