#6523 [Engine] Windows NUMA in Strata: Why Our 40-Core Server Initially Used Only 20 CPU Workers
open · @BlackKawa0808 · 0 Kommentare · Auf GitHub
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Beschreibung
*A technical follow-up report on the HPE ML350 Gen10, DeepSeek V4.1 Flash, and a separate NUMA test engine* **Status:** Published investigation; addendum updated for upstream Strata 0.2.18 and the experimental Windows NUMA patch. A performance comparison against a genuinely newer official DSV41 engine remains pending. **Investigation period:** November 19–21, 2026. **Primary reference:** [ONE vs. TWO? Benchmarking DeepSeek V4.1 Flash with Strata (#6123)](https://github.com/Niko1221/Strata/issues/6123) **Related investigation:** [24 GB vs. 48 GB: RTX 3090 Ti vs. Quadro RTX 8000 (#6298)](https://github.com/Niko1221/Strata/issues/6298) ## 1. Why We Investigated CPU Utilization More Closely Our earlier benchmarks showed that a large proportion of the DeepSeek expert weights had to be processed in the server's system RAM rather than GPU memory. This raised another question: Was the Windows engine actually using both Xeon processors for CPU expert computation? The initial checks revealed something unexpected. Although Strata Desktop detected both processors and all 40 physical cores, the running DeepSeek engine reported only **one NUMA part and 20 CPU-lane threads**. The remaining physical cores were not being incorporated into this workload accordingly. This report covers the identified cause, our isolated fix, verification against the running engine, and the measured effects. We will not repeat the original GPU and model benchmarks here. The linked reports above provide their test conditions and background. As in our previous investigations, I performed the tests on the server and verified the console output and measured results. ChatGPT (referred to below as “Chat”) assisted with investigation planning, scripts, source-code analysis, and documentation. Wherever possible, we checked our conclusions against actual output and the code we inspected. ## 2. Test Platform and Observed Problem | Component | Configuration | |---|---| | Server | HPE ProLiant ML350 Gen10 | | CPUs | 2 × Intel Xeon Gold 6242R, 40 physical cores / 80 logical processors in total | | NUMA | 2 nodes, 20 physical cores each; Windows processor groups 0 and 1 | | RAM | 768 GB installed | | Operating system | Windows Server 2025, Hyper-V enabled | | GPU used for this benchmark | NVIDIA Quadro RTX 8000, 48 GB VRAM | | Model | DeepSeek V4.1 Flash UD-Q8_K_XL | | Context length for comparative benchmarks | 8,192 tokens | An RTX 3090 Ti is also installed in the server, but it was **not** used as a second inference GPU for these NUMA comparisons. Our original engine reported **20 CPU-lane threads and one NUMA part**. The process was permitted to use both Windows processor groups, but the observed thread affinity and CPU load of the original build were concentrated in one group. The problem was therefore not merely how the available hardware was displayed, but how it was actually incorporated into the CPU computation path. ## 3. Root Cause Identified in the Windows Code Our source-code investigation led to `src/dsv41/ds4_os.hpp`. In the Windows implementation we examined, `node_core_threads()` considered only **processor group 0** when detecting the CPU/NUMA topology. Thread affinity was also set using `SetThreadAffinityMask`, without explicitly specifying the intended Windows processor group. On our dual-socket server, each Xeon belongs to a separate processor group. This explains why the engine constructed a CPU-lane topology limited to effectively 20 cores, even though 40 physical cores were available. **Scope:** We investigated this behavior on our particular Windows Server configuration. We are not claiming that every Strata version or every multi-socket Windows system is affected in exactly the same way. ## 4. Our Separate NUMA-v1 Build We did not overwrite the original engine. Instead, we copied the source tree, applied a Windows-specific patch, and compiled a separate test build: | | Original | NUMA-v1 test | |---|---|---| | Source directory | `G:\Strata\Source\Strata-main` | `G:\Strata\Source\Strata-NUMA-Test` | | Engine binary | `G:\Strata\Build\strata-dsv41.exe` | `G:\Strata\Build-NUMA-Test\strata-dsv41.exe` | | Modification | Unmodified reference build | Windows NUMA detection and processor-group-aware thread pinning in `ds4_os.hpp` | | Binary SHA-256 | `5FB22AE6D7957A19470F1ECB4AA4FD19C8FAAA756FF6ED773CE6F7AD612D0765` | `ACB160159D5787C9AD0B8760C2675BFA05FA0C5888D3213CECED514A76F8C7CA` (measured at build time) | The patch encodes Windows CPU identifiers across processor groups, detects both NUMA nodes, and uses `SetThreadGroupAffinity` for thread affinity. It was intended to leave other model and GPU computation paths unchanged. We compiled the engine as a separate Release build using Visual Studio 2022/MSVC, Ninja, and CUDA 12.8. The build completed with exit code 0. This patch is **a tested approach for this particular two-NUMA-node platform**, not yet a universally validated Windows fix. ## 5. Verification on the Running Engine The NUMA-v1 build successfully started with the Quadro RTX 8000 and the DeepSeek model. Its initialization log reported: - **2 NUMA nodes**, each with 20 physical cores; - **40 CPU-lane threads** and 2 NUMA partitions; - **13,708 CPU-resident experts**, with approximately 257.7 GB copied into RAM; - **1,652 GPU-resident experts** in the comparable 8K-context configurations. We also inspected actual thread affinity. **20 individually pinned threads** were assigned to each of Windows processor groups 0 and 1. During inference, both NUMA nodes showed substantial CPU activity. This confirmed that the test build was not merely detecting both processors, but also using them in the computation workload. The separate original engine, by contrast, still reported **1 NUMA part and 20 CPU-lane threads** in the return tests. ## 6. Performance Comparison We used the same Quadro RTX 8000, the same model, and the same adapted API benchmark script (`strata_bench_4080.py`). In the comparative benchmark blocks, the context length was 8,192 tokens; the effective chunk limit was **32,768 tokens**, as established from the later initialization log message. Each block consisted of nine short decode requests generating 100 tokens each, plus three fresh 4K prompts containing **4,108 actual input tokens**. Values below are medians per benchmark block. No requests failed. | Engine / benchmark block | Prefill, tok/s | Decode, tok/s | Time to first token, short request | |---|---:|---:|---:| | Original return test 01 | 197.1 | 16.1 | 0.87 s | | Original return test 02 | 202.1 | 15.4 | 0.88 s | | NUMA-v1 test 01 | 202.2 | 16.3 | 0.66 s | | NUMA-v1 test 02 | 207.3 | 17.8 | 0.60 s | | NUMA-v1 test 03 | 188.2 | 16.1 | 0.67 s | | NUMA-v1 test 04 | 204.5 | 16.6 | 0.68 s | | NUMA-v1 test 05 | 206.2 | 17.7 | 0.61 s | | NUMA-v1 test 06 | 210.4 | 17.7 | 0.61 s | **The most consistent observation:** Short-request time to first token was **0.60–0.68 s** with NUMA-v1, compared with **0.87–0.88 s** in the two original-engine return tests. Depending on the benchmark blocks compared, this represents an observed reduction in time to first token of approximately 22–32%. These measurements alone do not isolate the exact cause of the improvement. **Decode** often performed better with NUMA-v1, but results varied. **Prefill** ranges overlapped substantially; in particular, NUMA-v1 test 03 recorded only 188.2 tok/s, below the original-engine return tests. It would therefore be incorrect to claim that doubling the number of utilized CPU cores doubled performance. Across NUMA-v1 blocks 04–06, which were performed without reloading the model, median prefill increased from 204.5 to 210.4 tok/s. Median decode increased from 16.6 to 17.7 tok/s and then remained at that level. Cache effects, warm-up behavior, and other runtime factors remain possible explanations, not confirmed root causes. ## 7. NUMA Memory Locality: Additional Investigation A second important question was whether the two Xeons were predominantly processing their respective expert-weight partitions from local RAM. **Source-code path:** In the examined test engine, `lane_cfgs()` constructs per-NUMA-node CPU lists for the default configuration. `pools_for()` creates separate worker pools from those lists. `run_nodes()` dispatches task `f[X]` to `pools[X]`. When copied expert weights are used, the computation path accesses `ram_part[slot][X]` through `mx_part_ptr(..., X)`. The documented code path thus supports the intended mapping **matrix partition X → RAM partition X → worker pool X → CPU node X**. **Observed physical memory placement:** In one Strata process session, `VirtualQueryEx()` enumerated 5,762 virtual memory regions, including 690 larger committed private regions of at least 32 MiB (283.04 GiB in total virtual region size). We sampled 32 addresses from each of eight of the largest regions using `QueryWorkingSetEx()`. All **256 sampled pages** were valid, with no API errors. Within the sample, each individual region was associated exclusively with one of the two NUMA nodes (128 sampled pages on node 0 and 128 on node 1 in total). **Memory allocation path:** The normal Windows path uses `node_map(bytes, X, NH)`. An alternative huge-page path is selected automatically only under specific conditions. In the source code examined, `huge_free_bytes()` reports 0 on Windows, while `DS4_COPY_HUGETLB` was not set in the Strata JSON configuration or the checked PowerShell process, user, and system environments. We therefore expect the node-specific `node_map()` path for a normal startup. However, we did not directly inspect the environment block of the running engine process itself. **Limits of the evidence:** We did not map the eight sampled memory regions to specific expert matrices. Nor did we obtain hardware-counter measurements of local versus remote memory accesses or UPI traffic. Shared buffers, prefetching, GPU staging, and control threads can still cause cross-socket accesses. Accordingly, we **do not claim that every RAM access is NUMA-local**. ## 8. Additional Technical Observation, Not a Direct Benchmark Comparison A later NUMA-v1 model startup on November 21, 2026 used a context length of **262,144 rather than 8,192 tokens**. The log reported **13,724 out of 13,724 CPU experts** copied into RAM (258.0 GB, 2 NUMA partitions, 40 threads) and an effective chunk limit of 32,768 tokens. This confirms the later operating state, but the changed configuration means it **must not be treated as equivalent to the 8K-context performance benchmarks**. ## 9. Limitations, Reproducibility, and Preliminary Conclusion Our findings concern a specific Windows platform and the source revision we examined. The comparative benchmark series consist of practical repeated measurements, not a fully controlled statistical study. In particular, the number of CPU workers cannot be translated directly into a proportional performance gain. Despite these limitations, the main result is clear: **On our dual-Xeon system, the original engine used only one NUMA node with 20 workers for CPU expert computation. Our separate NUMA-v1 engine incorporated both Windows processor groups and demonstrably ran with 40 CPU-lane threads and two NUMA partitions.** The clearest practical change was a repeatedly shorter time to first token. This makes processor-group-aware Windows NUMA detection worth investigating on similar systems and in future Strata development. ## 10. Addendum: Status of the Official Developer Engine (November 21, 2026) Before attempting a third engine benchmark, we separately downloaded the official [Strata 0.2.13 source archive](https://github.com/Niko1221/Strata/releases/tag/v0.2.13) and compared its DeepSeek source directory, `src/dsv41`, against our unmodified original source tree. **Result: 51 files checked, 0 content differences**, based on per-file SHA-256 comparisons. This includes `ds4_os.hpp`, which is relevant to Windows NU
Mehr auf der Site
Links zu Install, Modellen, Releases.