#2456 [Benchmark] Snakes and Ladders in 3D: Comparing Strata Variations + Ollama and llama.cpp — RTX 5060 Ti 16GB + 96GB RAM

说明

**On my machine, Strata-served models produced substantially better games than the non-Strata setups I tested on the same application-building task.** My best Strata build scored **94.8/100**. The best scored non-Strata builds reached **65.1** for GLM-5.3-Flash via llama.cpp, **59.9** for Qwen3.6-35B-A3B coding via Ollama, and **33.3** for Qwen3-Coder-Next via llama.cpp.

This is a comparison of complete local setups—model, quantization, inference backend, context, and agent—not a controlled measurement of the inference engine alone. All full-task comparisons used **Claude Code as the harness**, with inference supplied by the selected local model. No Anthropic model performed the tested coding work. Claude helped create the benchmark and reviewed code separately.

> **Low results from the tested non-Strata setups:** Qwen3.6 coding via Ollama averaged **59.1/100**, GLM via llama.cpp averaged **52.0/100**, and Qwen3-Coder-Next via llama.cpp scored **33.3/100**. Their average game ratings were only **33.0, 19.5, and 18.0**, respectively. Every fully scored Strata game in this dataset finished above the best scored non-Strata result.

## Same task, different local setups

Every full-task run started from the same seed repository and specification: build a **3D snakes-and-ladders game** in TypeScript, three.js, and Vite, across six milestones. The agent had to produce a working application, not just answer coding questions.

The table includes only full-task builds with all three scoring components present. Averages pool the tested settings for each model; they are descriptive summaries, not matched-condition estimates.

| Local model / backend | Scored full-task builds | Mean function | Mean code | Mean game rating | Mean total | Best total |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Qwen3.8-Flash-Next UD-Q4_K_XL / Strata | 5 | 100.0 | 83.6 | 84.0 | **90.3** | 92.8 |
| Qwen3.8-Flash-Next UD-IQ4_XS / Strata | 4 | 100.0 | 81.5 | 85.5 | **90.1** | **94.8** |
| Qwen3.8-Flash-Next IQ3_S / Strata | 9 | 99.1 | 80.6 | 74.0 | **86.0** | 91.5 |
| Qwen3.8-Flash-Next Coder IQ1_M / Strata | 3 | 100.0 | 69.7 | 68.0 | 81.3 | 89.5 |
| Qwen3.8-Flash-Next Swift IQ3_XXS / Strata | 3 | 100.0 | 79.3 | 54.0 | 80.0 | 88.5 |
| **Non-Strata: Qwen3.6-35B-A3B coding / Ollama** | 2 | 91.6 | 42.0 | **33.0** | **59.1** | **59.9** |
| **Non-Strata: GLM-5.3-Flash E256 Q2 / llama.cpp** | 4 | 66.5 | 65.1 | **19.5** | **52.0** | **65.1** |
| **Non-Strata: Qwen3-Coder-Next UD-IQ4_XS / llama.cpp** | 1 | 44.2 | 34.0 | **18.0** | **33.3** | **33.3** |

The difference appeared in the finished applications as well as the code reviews. Ollama's Qwen runs scored fairly well on automated function checks, but much lower on code quality and my game ratings. The GLM runs were weaker in game quality, and some gave up or looped. The single Qwen3-Coder-Next run ended with substantially lower function, code, and game scores.

The gap was not just a comparison against Strata's single best result: IQ3_S averaged **86.0** over nine full-task builds, and both larger Unsloth variants averaged about **90**. In contrast, the two Ollama builds scored **59.9 and 58.4**. Even the best scored GLM build, **65.1**, remained below every scored full-task Strata build in this dataset; the lowest Strata total was **72.8**. This is a consistent separation within these saved runs, although the conditions and number of trials are unequal.

**Qwen3-Coder-30B-A3B-Instruct UD-Q4_K_XL was also tested via llama.cpp**, at 192K context. Its function score is 44.2 and my game rating is 18.0, but its code review is missing. It therefore has **no final total** and is excluded from the ranking above.

### Low non-Strata results: individual runs and score gaps

Even when comparing each non-Strata setup's **best scored run** with Strata's **best scored run**, the gaps were substantial:

| Non-Strata setup | Best total / 100 | Points below the best Strata build (94.8) |
| --- | ---: | ---: |
| GLM-5.3-Flash E256 Q2 / llama.cpp | **65.1** | **29.7** |
| Qwen3.6-35B-A3B coding / Ollama | **59.9** | **34.9** |
| Qwen3-Coder-Next UD-IQ4_XS / llama.cpp | **33.3** | **61.5** |

These are descriptive score gaps between the tested setups, not percentage improvements or isolated engine effects. Qwen3-Coder-30B-A3B remains unranked because its code score is missing.

| Model | Context | Effort | Build labels | Individual totals |
| --- | --- | --- | --- | --- |
| Qwen3.6-35B-A3B coding / Ollama | 256K | low | T, X | 59.9, 58.4 |
| GLM-5.3-Flash E256 Q2 / llama.cpp | 256K | low | AE, AI | 37.2, 50.2 |
| GLM-5.3-Flash E256 Q2 / llama.cpp | 256K | high | AF, AJ | 55.5, 65.1 |
| Qwen3-Coder-Next UD-IQ4_XS / llama.cpp | 256K | high* | AL | 33.3 |
| Qwen3-Coder-30B-A3B-Instruct UD-Q4_K_XL / llama.cpp | 192K | high* | AM | Not yet fully scored |

GLM's low-effort runs averaged **43.7**; high-effort runs averaged **60.3**. AE and AF gave up, AI looped, and AJ completed. Additional GLM max-effort attempts were stopped or hit an infrastructure error and have no final totals.

**Important qualification for GLM:** AE and AF each recorded **121 server errors** and **10 automatic resumes**. Their results reflect backend/session problems as well as the generated code, so the four-run GLM mean must not be interpreted as a clean measurement of model ability. Later runs AI and AJ recorded zero server errors and scored **50.2 and 65.1**. AJ is marked complete but its telemetry also flags a loop; these fields describe different aspects of the run.

*The Qwen3-Coder models are marked `high` in the lab configuration, but the backend identifies both as non-thinking models. This label should not be read as equivalent to a thinking model's high-effort setting. GLM E256 Q2 is the pruned, 2-bit expert variant recorded by the lab, not an unquantized GLM result.*

## Hardware and environment

- Intel Core i9-13900K
- 96 GB installed RAM (approximately 94 GiB available to the host)
- NVIDIA RTX 5060 Ti, 16 GB VRAM
- Strata running in Docker on Unraid
- Original Strata run: v0.1.40.3, commit `d5ea713`; Claude Code 2.1.280, Qwen Code 0.24.5, Pi 0.87.1, Ollama 0.40.2, as recorded in the benchmark notes
- Non-Strata models were served through Ollama or llama.cpp's `llama-server`; the local launcher pins `server-cuda-b11552`

The first Strata series ran on October 9–11, 2026; the saved dataset also contains later non-Strata comparisons. The results and screenshots here were retrieved on October 14, 2026.

The benchmark notes record stopping competing model servers and Plex/Jellyfin during the original Strata lab run to give it the GPU and RAM. I have not independently established identical background-load conditions for every later comparison, so that remains a reproducibility limitation.

## What the game task actually required

The specification deliberately combined rules, rendering, interaction, and polish. Finishing a rules module alone was not enough:

| Milestone | Required result |
| --- | --- |
| 1. Rules and test hook | Pure TypeScript rules, seeded randomness, forced dice sequences, a text interface, and rule tests |
| 2. 3D board | A correctly numbered 10×10 board, alternating tiles, lighting, soft shadows, an orbit camera, and resizing |
| 3. Die and tokens | A tumbling 3D die showing the actual roll, tokens hopping square by square, and sensible placement when tokens share a square |
| 4. Snakes and ladders | 3D ladder rails and rungs, curved snakes with heads, and movement along their paths |
| 5. Game flow | Configurable 2–4 human/computer players, start screen, turn banner, computer turns, winner celebration, and restart |
| 6. Polish | Camera follow, generated sound effects, mute and keyboard controls, phone layout, reduced motion, and no console errors |

The rules include nine fixed ladders and ten fixed snakes, an extra roll on a six, exact landing on 100 to win, and bouncing back when a roll overshoots. A bounce can land on a snake: **97 + 5 lands on 98, then slides to 78**. These edge cases help distinguish a plausible-looking board from a correct game.

The required `window.__game` hook exposes state and a promise-returning roll operation. URL parameters control the random seed, dice sequence, player count, and instant animations. This lets the external grader play repeatable games and compare actual state transitions, rather than trusting the agent's final message.

The seed contained installed tooling and the specification, rather than a completed game to edit. Packages were fixed—TypeScript, Vite, three.js, Vitest, Playwright, and ESLint—with no network access to install additional dependencies or download art. Audio had to be generated with the Web Audio API.

## Method and scoring

I first checked three harnesses against the same Strata IQ3_S model at 256K context and high effort. This check covered **milestones 1–2 only**, rather than the complete game:

| Harness | Automated function/code score | Wall time |
| --- | ---: | ---: |
| Claude Code | 92.7 | 18.4 minutes |
| Qwen Code | 94.2 | 84.1 minutes |
| Pi | 93.8 | 43.2 minutes |

All three passed the rule checks. Claude Code reached a similar automated score fastest in this check, so it was used for the remaining tests.

In this one harness check, Qwen Code took approximately **4.6×** Claude Code's wall time and Pi took **2.3×**. Their output-token totals were about **201K**, **94K**, and **49K**, respectively. This supports the harness choice for this experiment; it is not a general claim that Claude Code is always the fastest harness.

### How a full run proceeded

1. The lab selected a model, configured context window, effort, and backend, then prepared the local model server.
2. The agent started in a fresh sandbox from the common seed repository and task specification.
3. It was instructed to work through the milestones in order, run tests/type checks/lint/build before each milestone commit, and inspect screenshots after visual milestones.
4. A relay routed inference to the local backend and recorded requests, tokens, tool failures, errors, context use, compactions, and timings.
5. The external grader checked the generated application, and the lab retained the code, metrics, screenshots, and run outcome.
6. A blind code review and my game ratings supplied the remaining scoring components.

The orchestrator could send a fixed continuation message when an agent stopped early, up to ten times within the run's time limit. This was automatic continuation, not bespoke human debugging. Most configurations had an eight-hour limit; UD-Q4_K_XL and the tested GLM configuration had sixteen-hour limits. The exported `resumes` field is zero for the scored Strata runs and ten for GLM AE and AF.

Configuration selection was staged rather than an exhaustive sweep. IQ3_S served as the anchor; context and effort were explored, other Strata variants were added, and selected larger windows and higher efforts were requested afterward. Some low-scoring settings received a second run. The winner is therefore the best **observed** build in this search, not an estimate of the best possible configuration.

Full-task scores combine:

- **40% function:** hidden scripted gameplay checks, build/test checks, stability, and milestone completion.
- **30% code:** a blind Claude review against a fixed rubric covering architecture, readability, tests, correctness, and hygiene.
- **30% game rating:** my manual scores for gameplay, visuals, animation, controls, and bugs.

The dashboard's automatic score uses function and code only, weighted 4:3. **This report ranks by the total that includes my game rating.** The gallery rounds scores to whole numbers; the tables retain the exported one-decimal values.

### What each score measures

For a fully scored build, **total = 0.40 × function + 0.30 × code + 0.30 × game rating**. The winning build Z therefore scores `0.40 × 100 + 0.30 × 82.5 + 0.30 × 100 = 94.75`, displayed as **94.8**.

The function