#6298 [Benchmark] 24 GB vs. 48 GB: DeepSeek V4.1 Flash with Strata (RTX 3090 Ti vs. Quadro RTX 8000)
open · @BlackKawa0808 · 0 评论 · 去 GitHub 看
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
说明
*A follow-up benchmark investigation on an HPE ML350 Gen10* **Test period:** November 15–19, 2026 **Model:** DeepSeek V4.1 Flash UD-Q8_K_XL **Scope:** 14 benchmark runs per GPU (A–N) **Previous report:** [ONE vs. TWO? Benchmarking DeepSeek V4.1 Flash with Strata](https://github.com/Niko1221/Strata/issues/6123) ## Preface In my previous report, [“ONE vs. TWO? Benchmarking DeepSeek V4.1 Flash with Strata”](https://github.com/Niko1221/Strata/issues/6123), I investigated how my HPE ML350 Gen10 with a single Quadro RTX 8000 compared with an external system equipped with two of these graphics cards. This report builds on that investigation. In addition to the Quadro RTX 8000 with 48 GiB of VRAM, my server also contains a GeForce RTX 3090 Ti with 24 GiB of VRAM. That led to a new question: **How do the two GPUs compare when each runs the same large AI model independently on the same server?** Unlike the original comparison, we could keep the processors, system memory, operating system, and engine binary constant this time. The GPUs were tested separately. There was no combined inference and no pooling of their VRAM. Once again, this investigation was carried out in collaboration with **ChatGPT (hereafter referred to as “Chat”)**. I was responsible for carrying out the tests, configuring the hardware, and independently checking the measurement results. Chat assisted with experimental design, data analysis, technical interpretation, and documentation. We cross-checked the conclusions against actual console output, raw measurements, and engine logs. This report focuses on the new results. A detailed account of the original methodology, Strata engine, and earlier investigation is available in the [first report](https://github.com/Niko1221/Strata/issues/6123). ## 1. Test System and Methodology | Component | Configuration | |---|---| | Server | HPE ProLiant ML350 Gen10 | | Processors | 2 × Intel Xeon Gold 6242R, 40 cores / 80 threads in total | | System memory | 768 GiB DDR4-2666 | | Operating system | Windows Server 2025 Datacenter | | GPU 1 | GeForce RTX 3090 Ti, 24 GiB VRAM | | GPU 2 | Quadro RTX 8000, 48 GiB VRAM | | GPU connection | PCIe 3.0 ×16 for each card | | Model | DeepSeek V4.1 Flash UD-Q8_K_XL | | Context length | 8,192 tokens | | Thinking / vision | Disabled | | Sampling temperature | 0, explicitly set in each benchmark API request | **Both benchmark series used the same Strata DSV41 engine binary**, verified by matching SHA-256 hashes (`5FB22AE6D7957A19470F1ECB4AA4FD19C8FAAA756FF6ED773CE6F7AD612D0765`). The inference engine binary was not changed between the two GPU test series. **Sampling-temperature verification (November 19, 2026):** During a later review, we noticed that the model's saved Strata Desktop configuration did not initially contain an explicit `sampling.temperature` entry. We therefore checked the retained benchmark scripts and the local Strata server and DeepSeek engine source files. The benchmark HTTP requests explicitly sent `"temperature": 0`; the examined server code gives this request value precedence over saved defaults and the DS4 engine uses temperature 0 as its default for an individual request. This supports the originally documented greedy-sampling setup for both benchmark series. We have not independently established a byte-for-byte match between the examined server Python source and every historical server session. A subsequent RTX 3090 Ti control run with temperature 0 saved in Desktop produced different throughput figures, but did not isolate temperature as their cause; it is not part of the A–N comparisons below. The tests ran through the Strata Desktop API using the same underlying benchmark methodology: - Nine short chat requests, each generating 100 tokens, to measure decode throughput. - Three fresh longer prompts, each generating up to 24 tokens, to measure prefill throughput. - Medians as the primary comparison metrics; individual measurements are recorded in the raw data. - Actual prompt-token counts reported by the model, rather than relying only on target lengths. - No reuse of previously processed prompt tokens in the long-prompt tests. For the targeted boundary tests, variants of the Desktop-compatible benchmark script were used with adjusted target prompt lengths. With an 8,192-token context, the 16K and agent subtasks were skipped as intended. GPU assignment was checked using the Strata configuration, process environment, engine logs, and GPU telemetry. **Only one GPU was selected for model inference in each test.** The other GPU remained disabled in Strata. Automatically generated hardware headers sometimes still listed both physically installed GPUs. ## 2. Direct Performance Comparison The following table covers the complete A–N series. Prefill and decode values are medians in **tokens per second**. The chunk column gives the *effective* setting for the RTX 3090 Ti / RTX 8000, respectively. | Test | Prompt tokens | Chunk 3090 / 8000 | RTX 3090 Ti prefill | RTX 8000 prefill | Difference | Decode 3090 / 8000 | |---|---:|---|---:|---:|---:|---| | A | 4125 | 4096 / 32768 | 219.7 | 205.3 | +7.0% | 15.1 / 15.3 | | B | 4125 | 4096 / 4096 | 220.4 | 179.1 | +23.1% | 14.7 / 16.7 | | C | 4125 | 4096 / 32768 | 218.0 | 207.2 | +5.2% | 14.8 / 16.8 | | D | 4125 | 4096 / 4096 | 217.4 | 175.6 | +23.8% | 14.8 / 16.3 | | E | 3529 | 4096 / 4096 | 261.2 | 203.9 | +28.1% | 15.1 / 15.6 | | F | 3529 | 4096 / 32768 | 229.7 | 197.7 | +16.2% | 14.3 / 16.7 | | G | 4028 | 4096 / 32768 | 246.6 | 211.6 | +16.5% | 14.1 / 16.0 | | H | 4028 | 4096 / 4096 | 285.2 | 204.8 | +39.3% | 15.4 / 16.8 | | I | 4078 | 4096 / 4096 | 290.1 | 209.2 | +38.7% | 15.5 / 16.1 | | J | 4108 | 4096 / 4096 | 218.2 | 179.0 | +21.9% | 15.5 / 16.0 | | K | 4108 | 32768 / 32768 | 200.5 | 202.0 | −0.7% | 11.9 / 16.8 | | L | 4108 | 32768 / 32768 | 207.0 | 204.6 | +1.2% | 11.8 / 16.1 | | M | 4108 | 4096 / 4096 | 219.7 | 178.7 | +22.9% | 15.4 / 16.3 | | N | 4108 | 4096 / 4096 | 220.4 | 182.9 | +20.5% | 15.5 / 15.6 | *The percentage difference is relative to the RTX 8000's prefill median. Matching test letters denote corresponding stages of the investigation, not necessarily identical internal memory layouts. These percentages are therefore descriptive system-level comparisons, not isolated measures of an architectural GPU performance advantage.* ## 3. Main Findings ### 3.1 The RTX 3090 Ti Achieves Higher Prefill Throughput In almost all test runs, the RTX 3090 Ti achieved higher prefill throughput than the Quadro RTX 8000. Test **I**, with 4,078 prompt tokens and an effective chunk size of 4,096, is particularly notable: - **RTX 3090 Ti:** 290.1 tok/s - **Quadro RTX 8000:** 209.2 tok/s - **Advantage:** 38.7% This demonstrates that, under a suitable test configuration, the RTX 3090 Ti can process input prompts substantially faster despite having half as much VRAM. Decode throughput tells a different story, however. The RTX 8000 often achieved somewhat higher values. Faster prefill does not automatically mean faster continuous response generation. ### 3.2 Both GPUs Show a Drop at the 4,096-Token Boundary We deliberately investigated prompt processing immediately below and above the effective 4,096-token chunk boundary. | GPU | 4,078 tokens | 4,108 tokens | Change | |---|---:|---:|---:| | RTX 3090 Ti | 290.1 tok/s | 218.2 tok/s | **−24.8%** | | Quadro RTX 8000 | 209.2 tok/s | 179.0 tok/s | **−14.4%** | On both GPUs, prefill throughput fell sharply even though the prompts differed by only 30 tokens. The results are consistent with an additional processing step when the chunk boundary is crossed. **However, internal chunk execution was not directly instrumented.** The exact mechanism therefore remains a plausible explanation rather than a fully proven one. ### 3.3 More VRAM Changes Expert Residency Perhaps the most important new observation concerns memory management. On the RTX 3090 Ti, increasing the chunk size changed more than prompt processing. It also left the card with no GPU-resident experts. | Parameter | RTX 3090 Ti, chunk 4096 | RTX 3090 Ti, chunk 32768 | RTX 8000, chunk 32768 | |---|---:|---:|---:| | Physical VRAM | 24 GiB | 24 GiB | 48 GiB | | GPU-resident experts | 307 | **0** | 1652 | | GPU expert weights | 6.52 GiB | 0 GiB | 31.81 GiB | | GPU layer weights | 15.51 GiB | 15.51 GiB | 15.51 GiB | While the RTX 8000 could still keep 1,652 experts in VRAM at chunk 32,768, the RTX 3090 Ti had to process all experts through the CPU/system-RAM path. The model was still partly running on the RTX 3090 Ti. **This was not CPU-only inference.** This difference illustrates why a larger chunk size does not automatically produce better performance. ### 3.4 The J–N Cross-Check Confirms the Configuration Difference To check this behavior, we switched between chunk configurations while keeping the prompt length fixed at **4,108 tokens**: | Test | Chunk | GPU experts | RTX 3090 Ti prefill | RTX 3090 Ti decode | Long-prompt TTFT | |---|---:|---:|---:|---:|---:| | J | 4096 | 307 | 218.2 | 15.5 | 18.857 s | | K | 32768 | 0 | 200.5 | 11.9 | 20.523 s | | L | 32768 | 0 | 207.0 | 11.8 | 19.875 s | | M | 4096 | 307 | 219.7 | 15.4 | 18.739 s | | N | 4096 | 307 | 220.4 | 15.5 | 18.661 s | *GPU expert counts for L and N were carried forward from the most recently verified initialization of the engine, which remained running without a reported configuration change.* After switching back to chunk 4,096, both prefill and decode throughput returned to nearly their previous levels. The relationship between chunk configuration, expert residency, and performance was thus observed repeatedly and was reversible in this test series. **We did not isolate how much of the difference resulted from chunk processing itself and how much from the changed expert residency.** Without further instrumentation, runtime effects also cannot be ruled out completely. ## 4. Interpretation and Limitations Compared with the previous report, this comparison has an important methodological advantage: both GPUs were tested on the same server platform with the identical engine binary. Nevertheless, it is not a fully isolated hardware comparison. Different VRAM capacities affect how much of the model Strata can keep on the GPU. Also, some tests with the same letter used different automatically selected effective chunk sizes. GPU clock behavior and runtime conditions were not continuously recorded throughout every benchmark either. It would therefore be unjustified to claim that the RTX 3090 Ti is generally faster by any specific percentage. These measurements apply to the model, quantization, Strata engine, and documented test configuration used here. They cannot automatically be generalized to other models or inference engines. Complete individual measurements, TTFT data, and engine-log mappings are preserved in the archived A–N benchmark records. This summary does not replace those raw results. ## 5. Conclusion and Possible Follow-Up Investigation The results show that **24 GiB of VRAM can be sufficient for high prefill throughput**, even when a large portion of the model resides in system RAM. At the same time, they show that an unfavorable chunk configuration can have a substantial performance impact. For our 4,108-token test on the RTX 3090 Ti, the tested 4,096-chunk configuration performed better than 32,768. By contrast, the Quadro RTX 8000 benefited from the larger chunk boundary at the same prompt length and could still retain many GPU-resident experts. These findings lead to a new question: **Could Strata tune chunk size more effectively based on available VRAM and the GPU expert residency required, rather than relying primarily on fixed sizes across different memory configurations?** For a possible follow-up investigation, we would like to compare **different chunk sizes on the two GPUs already tested**, the RTX 3090 Ti with 24 GiB and the Quadro RTX 8000 with 48
本站相关内容
相关页面的快捷入口。