#5690 [Benchmark] add synthetic two-request reproduction for shared Qwen repetition (#606)

closed · @midhatn · 0 评论 · 去 GitHub 看

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDASecurityWindows

说明

## Problem and result
A fixed 61,000-token coding request preceded by a 64,000-token non-thinking request can intermittently terminate with an empty answer after 256 consecutive digit `0` tokens in reasoning. This shared failure was observed across both CUDA 12 native engines 0.1.58 (`0fe4ae5`) and 0.1.59 (`712fec6`) on 2× Tesla V100-SXM2-16GB with Qwen3.8-Flash-Next UD-IQ2_XS. It is not an established new 0.1.59 regression.

This adds a self-contained, sanitized synthetic two-request reproduction bundle under `bench/repros/issue606-v100-repeat/`. It includes exact wire request fixtures, verified SHA-256 manifests, reproduction documentation, and a standalone Python standard-library client (`reproduce.py`) that checks response protocols and text properties without ever executing model output.

## Related discussion
Existing issue https://github.com/Niko1221/Strata/issues/606 and observation comment https://github.com/Niko1221/Strata/issues/606#issuecomment-6751284700.

## Approach
Provide a minimal deterministic client and synthetic request fixtures to enable reproducible investigation of post-64K repeated-token failures without private conversations, user credentials, or host-specific paths:
- `01-long-64000.json` (64,000 input tokens, non-thinking, 512 completion tokens).
- `02-coding-61000-budget2999.json` (61,000 input tokens, thinking enabled, explicit budget 2,999, 4,000 max tokens).
- Standard-library client (`reproduce.py`) handles HTTP streaming, validates strict protocol and accounting bounds, rejects cached inputs, and safely counts character repetition without executing code.
- Engine code, serving paths, and default configurations are completely untouched.

## Validation

**Passed:**
- `python -m unittest test_reproduce.py` passed all 5 tests (0.005 s) on Windows with Python 3.11, verifying wire hashes, alias override scoping, credential security rejection, mock non-execution guarantee, and protocol validation rules.
- `git diff --check origin/main..evidence/qwen-v100-repeat-0158-0159` passed with no whitespace or formatting errors.
- Device observations: reproduced the shared 2,198 API output / 2,199 native generated token failure across 3 attempts on 2× V100-SXM2-16GB (0.1.59 new frontend, 0.1.59 old frontend, and repeat 0.1.58 old frontend), with zero ECC errors, zero thermal counter events, and clean process exits.

**Failed:**
- None observed in test suite or validation checks.

**Not run:**
- Native compilation builds (CUDA, HIP, SYCL, Metal): no native C++/CUDA source was modified.
- Full hardware test matrix across different GPU architectures (tested only on dual V100-SXM2).

## Limits
This is a synthetic reproduction bundle and diagnostic harness only; it does not diagnose or resolve the underlying engine/kernel/model state causing the repetition, nor does it establish whether floating-point values remain finite.

本站相关内容

相关页面的快捷入口。