#2084 System hard-freezes during long-prompt prefill on NVIDIA Pascal = driver bug 6869496 (PMU halt from PCIe link retraining) — verified workaround: NVreg RmDisableGen2LinkRetraining=1

说明

## Hard system freezes during long-prompt prefill on NVIDIA Pascal — driver bug 6869496 (PMU halt from PCIe link retraining), with a verified driver-registry workaround

Not a Strata bug — but a Strata workload pattern that **reliably triggers** an NVIDIA driver bug, and a fix that may save other users of older NVIDIA GPUs a lot of pain. Suggest adding it to the troubleshooting docs.

### Setup
- GTX 1080 Ti (GP102), nvidia 575.64.05 proprietary, kernel 6.18 (also reproduced on 580.178.04 / 580.159.04, kernel 7.2)
- Strata serving a ~46k-token conversation (IQ3_S MoE, experts streaming over PCIe)

### Symptom
The machine **hard-freezes** mid "reading the prompt" (the sustained prefill phase), requiring a power cycle — MTBF 15–50 min under load, never while idle. No kernel messages survive (journal cut off before the disk flush); it looks exactly like "running Strata freezes my computer" with nothing to go on.

### Root cause (as far as we could take it)
Streaming kernel netconsole over UDP caught the death:

```
NVRM: GPU at PCI:0000:84:00: GPU-611b1183-...
NVRM: Xid (PCI:0000:84:00): 62, 00002810 ...   ← PMU_HALT_ERROR
```

Xid 62 = the GPU's PMU microcontroller halted — NVIDIA-tracked internal bug **6869496** (forums.developer.nvidia.com threads 384008, 385543 — the latter is the same GPU, works fine under Windows).

Reverse-engineering both driver binaries (Linux `nvidia.ko` 575.64.05 vs Windows `nvlddmkm.sys` 576.52) showed:

- PCIe **link speed switching is executed on the PMU** (`RMPcieGenSwitchOnPmu` in both)
- The registry key `RmDisableGen2LinkRetraining` is **live/wired in the Linux driver's registry table** (two descriptor entries) but a **dead string in the Windows build** (no table entry) — i.e. the legacy dynamic link-retraining machinery is still active on Linux builds but effectively retired on Windows. Consistent with Windows users not seeing this.

Mechanism: bursty prefill traffic → the RM/PMU oscillates the PCIe link between Gen3 and low speed for power saving → frequent PMU link-retrain handshakes → hits the bug → PMU halts → GPU dead → system frozen.

### Verified workaround (A/B tested)

`/etc/modprobe.d/nvidia.conf`:
```
options nvidia NVreg_RegistryDwords="RmDisableGen2LinkRetraining=1"
```

This pins the link at its max negotiated speed (Gen3, **no bandwidth loss**) and stops the PMU from doing link-speed transitions entirely. With everything else identical, crashes stopped completely:

| config | result |
|---|---|
| 575 + clock lock + power cap + new BIOS + CPU microcode + PC6/ASPM fixes | froze at 4.5 min, then at 35 min |
| same + `RmDisableGen2LinkRetraining=1` | 40-min soak with continuous 40k-token prefills: 0 freezes; real load since: 0 freezes |

(The clock lock `PerfLevelSrc=0x2222` and power-limit changes were **not** sufficient — the retraining key was the discriminating variable.)

### Suggestion for the project
Consider documenting this under a "system freezes while reading a long prompt (NVIDIA)" entry: the user-visible symptom points straight at Strata, the real culprit is the NVIDIA driver + old-GPU PMU bug, and the fix is one registry line. Possibly worth linking NVIDIA threads 384008 / 385543 and bug 6869496 so affected users can find it by searching either direction.

Happy to provide the netconsole logs and the driver-analysis notes if useful.