Pull requests / #2618

#2618 serve: /v1/models meta carries llama-server's model facts (n_params, …

open · @Pix13 · 0 Kommentare · Auf GitHub

Server & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Beschreibung

## Title

serve: /v1/models meta carries llama-server's model facts (n_params, size, ftype, ...) from the GGUF headers

Issue: none found (searched 2026-10-15: n_params, ftype, /v1/models meta). Related: #1882 (other context keys, same
dict; the two merge without overlap).

## Summary

llama-server's `/v1/models` gives each model a `meta` with `n_params`, `size`, `ftype`, `n_vocab`, `n_ctx_train`
and `n_embd`. Strata gives only `n_ctx`. Proxies and dashboards that read llama-server's model list (for example
Olla's Models panel) show the parameter count, the quantisation and the size on disk from these keys, and leave
them blank for a Strata endpoint.

This change reads them from the `--native` GGUF's headers (`tools/gguf_reader.py`, no tensor data), over every
shard of a split model, the first time `/v1/models` is asked for, and keeps them:

- `n_params`: the elements of every tensor in every shard
- `size`: the shards' bytes on disk
- `ftype`: llama.cpp's `llama_model_ftype_name()` label of `general.file_type` (`"Q8_0"`, `"IQ3_S - 3.4375 bpw"`)
- `n_vocab`, `n_ctx_train`, `n_embd`: `tokenizer.ggml.tokens`, `<arch>.context_length`, `<arch>.embedding_length`

Only the facts found are given. A missing shard gives no `n_params`/`size` (rather than a wrong one); an engine
without a GGUF (pack only, the mock engine) or a GGUF that cannot be read gives the old `meta`, and an unreadable
file is said once on stderr. `meta.n_ctx` is unchanged and stays the engine's.

What it leaves alone: the engine, every other endpoint, requests. On opt-in: the change only adds keys to a dict
whose existing keys keep their values; llama-server always sends them, so I did not put them behind a setting.
Happy to add one if you prefer.

## Measured

Header read, on Rocky 9.8 (files in the page cache):

| model | shards | read | n_params | size | ftype |
|---|---|---|---|---|---|
| huihui Qwen3.8-Flash-Next Q8_0 | 6 (188 GB) | 0.23 s | 176943899520 | 188225033440 | `Q8_0` |
| ISTA-DASLab GSQ-RCO IQ3_S | 2 (84 GB) | 0.74 s | 176943899520 | 83617662656 | `IQ3_S - 3.4375 bpw` |

Both run with this change since 2026-10-15 (2x RTX 3090 and a V100); Olla shows `177B` / `Q8_0` / `175.30 GB`.

## Test

- `serve/test_server.py`, new `ModelMeta`: a two-shard GGUF written by the test (named by either shard), a missing
  shard, no GGUF, an unreadable GGUF (said once), and `/v1/models` with an alias.
- `python -m unittest discover -s serve -p 'test_*.py' -t .`: 789 tests, 2 failures that fail on `main` too
  without this change (`test_keepawake ... early_close`, `test_responses.NoLeakedSampler`, timing-dependent here).
- Not tested: Windows, macOS/Metal, a non-split single-file GGUF on a real engine (covered only by the unit test).

Mehr auf der Site

Links zu Install, Modellen, Releases.