ByteDance Seed Team Paper Reveals Why DeepSeek Struggles with Long Contexts

Large language models often use chunked KV cache compression when handling long texts. It packs windows of tokens into fewer cache entries, cutting memory and attention costs in long-context reasoning.

ByteDance's Seed team shared a paper on arXiv. It says this compression adds a new position factor called token phase, meaning where a token sits within the compressed window. The team tested models like DeepSeek-V4-Flash and DeepSeek-V4-Pro. The same information can be easy to find at one phase but hard at another. They call this phase sensitivity.

The study shows that in large open-weight models using such compression, long-context retrieval accuracy can differ by up to 40 percentage points across phases. Average benchmark scores may hide this periodic weakness, offering a new way to explain unstable long-text performance.