#6118 [Engine] Cold 50K prefill regression: 0.1.52 → 0.1.59 → 0.1.60 on RTX 3060 12GB (−16.5%)
open · @pieterst67 · 3 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsLinux
说明
## Summary Cold-cache prefill throughput has progressively decreased between Strata engine versions 0.1.52, 0.1.59 and 0.1.60 on my RTX 3060 12GB system. Decoding performance remains approximately unchanged. Most importantly, **rolling back from engine 0.1.60 to 0.1.52 immediately restored the original prefill performance**, without changing the hardware or model. | Engine | Runs | Mean prefill | Change vs. 0.1.52 | |---|---:|---:|---:| | 0.1.52 | 3 | 942.7 tok/s | Baseline | | 0.1.59 | 2 | 839.5 tok/s | -10.9% | | 0.1.60 | 3 | 787.3 tok/s | -16.5% | | 0.1.52 (rollback) | 1 | 941 tok/s | -0.2% | At 50,136 prompt tokens, engine 0.1.60 requires approximately **10.5 seconds more prefill time** than engine 0.1.52. Prefill measurements are highly consistent within each engine version. ## Hardware and operating system - System: HP Z2 G8 Tower - CPU: Intel Core i7-11700 - GPU: NVIDIA GeForce RTX 3060, 12 GB VRAM - RAM: 64 GB - OS: Ubuntu 26.04.1, x86-64 - Engine platform: `linux-x86_64-cuda-cuda13` - Model storage: local SSD ## Strata versions Engines tested: - 0.1.52 - 0.1.59 - 0.1.60 Software versions: - Initial benchmarks: Strata CLI 0.2.13.1, Control 0.1.59.1 - Engine 0.1.60 benchmarks: Strata CLI 0.2.15, Control 0.1.61 - Final rollback to engine 0.1.52: Strata CLI 0.2.15, Control 0.1.61 The rollback test is significant because engine 0.1.52 returned to approximately 941 tok/s even with the newer CLI and Control installed. ## Model and configuration - Model: Qwen3.8-Flash-Next IQ3_S - Format: split GGUF - Context: 131,072 tokens - KV cache: int8 - Expert cache: auto - Prefill: auto - Parallel requests: 1 - Vision: enabled, CPU processing - API: OpenAI-compatible endpoint - Server: headless `strata serve`, managed by systemd - Client: Pi coding agent Environment variables: ```ini STRATA_ROUTE_PRIOR=0 STRATA_ROUTE_TAIL_SKIP=0 ``` Both routing optimizations were disabled throughout the benchmarks. ## Steps to reproduce 1. Select engine 0.1.52. 2. Start Strata with the configuration above. 3. Restart the server to ensure a cold prompt cache. 4. Send a request from an existing Pi coding session containing approximately 50K tokens. My test message was `Hi`. 5. Confirm that Strata reports 50,136 prompt tokens and 0 reused tokens. 6. Record prefill throughput, decode throughput and total duration. 7. Repeat after restarting the server. 8. Switch to engine 0.1.59 and repeat the measurements. 9. Switch to engine 0.1.60 and repeat. 10. Roll back to engine 0.1.52 and repeat once more. All recorded requests had exactly 50,136 prompt tokens and 0 reused tokens. The same nominal Pi session and test message were used. However, I have not independently verified that the serialized request payload was byte-for-byte identical across every test. ## Individual benchmark results All runs used 50,136 prompt tokens with zero cache reuse. | Engine | Run | Output tokens | Prefill tok/s | Decode tok/s | Duration | |---|---:|---:|---:|---:|---:| | 0.1.52 | 1 | 303 | 942 | 28.8 | 63.8 s | | 0.1.52 | 2 | 273 | 942 | 28.1 | 63.0 s | | 0.1.52 | 3 | 57 | 944 | 27.5 | 55.2 s | | 0.1.59 | 1 | 297 | 837 | 27.6 | 70.7 s | | 0.1.59 | 2 | 267 | 842 | 29.8 | 68.5 s | | 0.1.60 | 1 | 250 | 787 | 29.9 | 72.0 s | | 0.1.60 | 2 | 325 | 787 | 28.2 | 75.2 s | | 0.1.60 | 3 | 268 | 788 | 29.0 | 72.9 s | | 0.1.52 (rollback) | 1 | 338 | 941 | 29.2 | 64.8 s | ### Prefill observations Engine 0.1.52: - Original average: 942.7 tok/s - Range: 942-944 tok/s - Rollback result: 941 tok/s Engine 0.1.59: - Average: 839.5 tok/s - Range: 837-842 tok/s - Approximately 10.9% slower than 0.1.52 Engine 0.1.60: - Average: 787.3 tok/s - Range: 787-788 tok/s - Approximately 6.2% slower than 0.1.59 - Approximately 16.5% slower than 0.1.52 ### Decode observations Decode throughput remains broadly comparable across versions, approximately 28-29 tok/s. The regression therefore appears to affect prompt processing much more than token generation. Total request durations are not directly comparable because generated output lengths differ. ## Expected behavior Cold-context prefill performance should remain comparable to engine 0.1.52, or improve with newer engine versions, under equivalent conditions. ## Actual behavior Prefill throughput decreases progressively: - 0.1.52: approximately 943 tok/s - 0.1.59: approximately 840 tok/s - 0.1.60: approximately 787 tok/s Switching back to 0.1.52 restores approximately 941 tok/s. The behavior is reproducible across multiple cold-cache runs. ## Additional observations The GPU expert-cache hit rate varies between requests, but the prefill throughput remains remarkably stable within each engine version. Engine 0.1.60 introduced changes to prompt processing, making it particularly interesting that its prefill throughput is lower than both preceding tested versions on this hardware. I have not established whether the regression is caused by new kernels, automatic tuning, memory transfers or another engine change. ## Questions for the developers 1. Were there changes to prefill kernels or automatic prefill tuning between engine 0.1.52 and 0.1.60 that could explain this regression on an RTX 3060 12GB? 2. Is there a recommended prefill setting or diagnostic option that would help isolate the cause? 3. Would a byte-identical API request replay or additional engine timing logs help investigate this? I can provide sanitized startup logs, effective engine arguments and further benchmark results if useful. Thank you for investigating.
本站相关内容
相关页面的快捷入口。