#2825 Report: Qwen3.8-Flash-Next IQ3_S works on dual V100-SXM2-16GB (sm_70) with driver 550 / CUDA 12.6 (0.1.44.1)

说明

This is a separate report from #971 (that one covers L40S vGPU on driver 550). Here: **dual Volta (sm_70)**, which I don't think has been reported working with 0.1.44.1 yet.

### Hardware

| | |
|---|---|
| GPU | 2× Tesla V100-SXM2-16GB (sm_70, 16 GB each, bare metal) |
| Driver | 550.144.03 (exposes CUDA 12.4, intentionally not upgraded) |
| CUDA toolkit | 12.6 (installed alongside 12.1, only the toolkit — driver untouched) |
| CPU | 2× Intel Xeon E5-2640 v4 (2 sockets, 24 threads total) |
| RAM | 64 GB |
| OS | Ubuntu 22.04 |
| Strata | upstream 0.1.44.1 |
| Model | Qwen3.8-Flash-Next IQ3_S (GSQ-RCO, 78 GB, 2 shards) |
| MTP | fetched from original Qwen checkpoint (~5 GB) |

### Setup command

```bash
STRATA_EXPERIMENTAL_SM60=1 \
./setup.sh --setup --family qwen --model IQ3_S --gpus 0,1 \
  --port 8085 --api-key test-v100 \
  --gguf-dir <repo>/IQ3_S \
  --cuda 12 --vision gpu --build --no-start --yes
### Ref

#971 — driver 550 / CUDA 12 on L40S vGPU. Same `--cuda 12` mechanism, different hardware.
Notes:

STRATA_EXPERIMENTAL_SM60=1 is required for sm_70 (engine built with -DSTRATA_EXPERIMENTAL_SM60=ON).

System cmake is 3.22.1, CMakeLists requires ≥ 3.24. Used the venv's cmake 4.4.3 via PATH=<repo>/.venv/bin:$PATH.

--cuda 12 picked /usr/local/cuda-12.6/bin/nvcc (the 12.6 toolkit, not the driver's 12.4 ceiling) and lowered the driver floor to 525, as documented.

Results
Layer split across GPUs [0, 1] works; both V100 busy, one strata process per card.

VRAM (text only): ~15.5–15.8 GB used on each V100.

VRAM (with --vision gpu):

GPU 0: 1428 MiB (strata-vision) + 14128 MiB (strata) = 15556 MiB

GPU 1: 306 MiB (strata-vision) + 15376 MiB (strata) = 15682 MiB

RAM: ~46.8 GiB experts loaded.

Expert cache: 94–98% hit rate on long agent runs.

Decode: 50–71 tok/s (text), 75–80 tok/s on 1700-token generations with vision.

Prompt eval: 69–86 tok/s on short prompts; ~3 s for a 10K-token prompt (~3300 tok/s prefill).

MTP: acceptance rate 69–71%.

KV int8, context 65536.

Calibration results (./setup.sh --calibrate)
Median of 3 rounds for each value:

PCIe share (--pcie-frac):

Value	tok/s
0	70.5
0.2	70.6 (best)
0.32	69.2
0.55	67.4
1.0	56.3
Draft floor (--spec-min-p):

Value	tok/s
0.3	67.5
0.5	69.7
0.7	71.2 (best)
CPU workers:

Workers	tok/s
3	60.2
5	63.3
6	64.4
7	62.0
11	67.7 (best of tested)
Expert tier (adaptive):

default: 66.8

every 1, 80 swaps, decay 0.97: 73.9 (best)

every 1, 160 swaps, decay 0.97: 72.0

Final: 73.9 tok/s (vs 70.5 before calibration).

Real workload (agent use)
After calibration, ran an agent with tool calling (ask_user, search_web, write_note, update_memory) over the OpenAI API. Context grew from ~10K to ~17K tokens.

Tokens generated	tok/s
2526	68.6
2841	64.7
3656	74.3
1721	66.6
Average ~69 tok/s. Expert cache hit rate 94–98%.

Tool calling, streaming, multi-step agent loops all work on 2× V100-SXM2-16GB.

Vision
After adding --vision gpu, the encoder (strata-vision) loads on both cards (see VRAM numbers above). A screenshot of a Russian-localized Total Commander was correctly described. Decode with vision: 75–80 tok/s on ~1700-token generations.

Pitfalls worth knowing for other Volta users
Fork Hardin22/Strata-DualGPU (Strata 0.1.38) does NOT build on CUDA 12. Its src/core/mtp.cpp uses CUDA 13 API signatures (cudaGraphAddNode with edge data, cudaStreamGetCaptureInfo with 7 args). Error is cannot convert cudaGraphEdgeData** to size_t*. This is an API mismatch, not "Volta unsupported".

In 0.1.38 MIN_DRIVER = 580 blocked startup even with STRATA_EXPERIMENTAL_SM60=1 — there is no --cuda flag and no conditional floor. Upstream 0.1.39+ fixed this with --cuda 12. Confirms what's written here.

Installing CUDA 12.6 via apt flips the update-alternatives cuda symlink to 12.6. If another service (e.g. llama.cpp) relies on /usr/local/cuda → 12.1, restore it: update-alternatives --set cuda /usr/local/cuda-12.1. Otherwise a restart of that service may load the wrong runtime.

--pool-workers 5 tip from setup is not universal. On this machine (2× Xeon E5-2640 v4, equal cores per socket), 5 workers is worse than 11 or even 6:

3 → 60.2

5 → 63.3

6 → 64.4

7 → 62.0

11 → 67.7

Calibration picked 11 as the best of the tested set. Worth mentioning in docs/OLDER_GPUS.md that the "fewer workers on dual-socket" tip is hardware-specific and can hurt on equal-core systems.

Prompt > context is rejected, not truncated. With --max-context 65536, a 77 187-token prompt returns 400: request exceeds the context window: ... requests are never truncated. That is by design, but worth documenting for users with long histories — they either trim the client-side history or rebuild with a larger context.

Thanks
--cuda 12 works out of the box on Volta with 0.1.44.1, including vision and agent tool calls. Great work on the older-GPU path.

Ref
#971 — driver 550 / CUDA 12 on L40S vGPU. Same --cuda 12 mechanism, different hardware.