#6544 [Engine] Fix O(N^2) scan in getline_fd and avoid full-line copy in BGEN/parse_i64_list
open · @YusenZhang0601 · 0 comentários · No GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Descrição
## Problem and result
On long-context `--serve` requests (100K–167K prompt tokens, where the `GEN`/`BGEN` line is ~0.7–1.2 MB of comma-separated token IDs), the engine's stdin reader and request parser perform repeated quadratic scanning and multi-megabyte string copies before prefill even starts:
1. `getline_fd` in `engine/src/program/generate.cpp` reads stdin in `4096`-byte chunks and calls `buf.find('\n')` starting from offset `0` on every chunk, scanning ~150 MB of already-inspected bytes for a 1.2 MB line ($O(N^2)$), and then copies the completed line into `st->lines` instead of moving it.
2. `BGEN` / `BGENI` admission rewrites the entire prompt line with `line = std::string("GEN 1") + e2`, duplicating the full megabyte string.
3. `parse_i64_list` copies the entire token string into a temporary `std::string`, copies it again into a `std::istringstream`, and allocates a per-token `std::string` before calling `std::from_chars`.
With this fix, stdin reading uses a `65536`-byte read buffer with a monotonic `scanned` cursor (strictly $O(N)$), moves completed lines into the queue, keeps `BGEN`/`BGENI` lines in place by pointing `endp` at `e2`, and parses token IDs in place directly with `std::from_chars` using the exact same delimiters and `[0, INT32_MAX]` validation.
On my RTX A6000 workstation (Windows native, dual EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS, 262K INT8 KV), measured on my `0.1.56` fork (`YusenZhang0601/[email protected]`, commit `a1c2c904139d5c6a028750141d7a75f35253b76a`), this alone reduced end-to-end host-side latency outside prefill/decode (`host_outside_ms`, ABAB median of 3 warm turns per context length) by **−27.40 ms (−20.3%) at 167K tokens**:
| Prompt length | Baseline `0.1.56` (`host_outside_ms`) | With this fix (`host_outside_ms`) | Delta |
|---|---:|---:|---:|
| 1K tokens | 32.28 ms | 28.46 ms | −3.82 ms (−11.8%) |
| 100K tokens | 88.18 ms | 72.89 ms | −15.29 ms (−17.3%) |
| 167K tokens | 135.06 ms | 107.66 ms | −27.40 ms (−20.3%) |
## Related discussion
none
## Approach
- **`getline_fd`**: Increase the stack chunk buffer from `4096` to `65536` bytes, track `size_t scanned` across reads so `buf.find('\n', scanned)` only inspects newly appended bytes, reset `scanned = 0` when emitting a line, and push `std::move(l)` into `st->lines`.
- **Zero-copy `BGEN`/`BGENI`**: Record `bgen_rewritten`, `bgen_img`, and `char* bgen_tail = e2` instead of rebuilding `line`. Downstream `GEN`/`GENI` checks (`refresh_file_tier`, `dc_learning`, validation, and `endp`/`max_new`) use `bgen_rewritten` directly, preserving exact control flow for `QUIT`, `SAVE`, `RESTORE`, `GEN`, `GENI`, `BGEN`, and `BGENI`.
- **In-place `parse_i64_list`**: Pre-reserve `slen / 5` entries when `slen >= 64` and scan `[s, s + slen)` with the exact same comma/whitespace delimiters (`','`, `' '`, `'\t'`, `'\n'`, `'\r'`, `'\v'`, `'\f'`) and `std::from_chars` `[0, INT32_MAX]` error checks as the `std::istringstream` path.
Compatibility: Wire protocol, accepted delimiters, error messages, and parsed token IDs are 100% identical to upstream. No new environment switches or CLI flags are added.
## Validation
**Passed:**
- `origin/main` (`6e38f588a`, engine `0.1.64`, commit `9557917b0` on `contrib/getline-fd-bgen`) native Windows x86_64 build (MSVC `14.37.32822`, CUDA `13.0.48`, `sm_86`, CMake + Ninja): `ninja strata` PASS (`17.4s`, `strata.exe` `64,886,272 B`, SHA-256 prefix `f390d5cd…`), `strata --version` -> `strata 0.1.64` (compile + `--version` smoke; there is no standalone unit-test target for `generate.cpp`).
- Historical ABAB measurement on my `0.1.56` fork (`YusenZhang0601/[email protected]`, `a1c2c904`, RTX A6000 48 GB `sm_86`, Windows WDDM, PCIe Gen3 x16, 2× EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS 262K INT8 KV, 0 thermal throttling, 0 CPU interference): 1K `32.28 -> 28.46 ms`, 100K `88.18 -> 72.89 ms`, 167K `135.06 -> 107.66 ms` (`-27.40 ms / -20.3%`); full-stack functional acceptance suite (262K context, vision, three protocols, tool calls, 4-way concurrent) 10/10 PASS.
**Failed:**
- none observed
**Not run:**
- End-to-end GPU ABAB re-benchmark on `0.1.64` (`origin/main` `6e38f588a`) — the latency table above comes from the identical patch on `0.1.56`.
- 24,574-line `kvq` comparison (this change only parses integer token IDs on the host before inference).
- Linux, HIP, SYCL, and Metal builds (I only have a native Windows CUDA `sm_86` workstation to compile on; `#if defined(_WIN32)` / POSIX `::read` branches remain structurally unchanged).
## Limits
- Only optimizes the C++ engine stdin reader and token list parser; Python-side prompt tokenization and CSV formatting remain unchanged (those are separate concerns).
- `out.reserve(slen / 5)` is a heuristic for typical 4–6 digit token IDs; shorter IDs still grow cleanly via `std::vector::push_back`.
No site
Links install, modelos, releases.