Pull requests / #2238

#2238 test: exercise uncached BPE paths in tokenizer parity checks

open · @mustafaerdinc · 0 comentários · No GitHub

Server & APIAMD / HIPNVIDIA / CUDA

Descrição

## Summary

Tokenizer parity checks can reuse cached piece IDs from a previous algorithm and pass without executing the requested scan, heap, or default path. Clear the cache between comparisons and add regressions that prove deliberately incorrect output is detected.

Related to the investigation in #1385. This test coverage change does not establish the cause or resolution of that intermittent crash.

## What changed

- Clear the piece cache before forced scan, forced heap, and default parity comparisons.
- Add three negative regressions covering incorrect scan, heap, and default output after cache population.
- Use an independent serial reference for shared-tokenizer workers. Exercise warmed-cache and uncached runs, verifying all eight uncached workers reach BPE.
- Remove the speculative interpreter-fault attribution from the test docstring. Production tokenizer code and runtime defaults are unchanged.

## Validation

- The three negative regressions fail before the fix and pass afterward; independent review found no blocking issue in this test-only change.
- After rebasing onto main at a0116b2, ran the following on macOS arm64 with CPython 3.14.8 and the model tokenizer files available: **24 tests passed, zero skips**, in 18.825 seconds.

```sh
STRATA_TOKENIZER=/path/to/tokenizer PYTHON_JIT=0 python -B -m unittest -v \
  serve.test_detok tools.test_strata_tokenizer tools.test_tokenizer_plain_spans
```

- No CUDA, HIP, or SYCL source is changed; no backend build is claimed.

## Extra Notes

The threaded wrapper proves test coverage, not the cause or resolution of #1385. The original-workload and minimal-reproducer investigation remains separate; this PR should not close that issue.

No site

Links install, modelos, releases.