Pull requests / #2618
#2618 serve: /v1/models meta carries llama-server's model facts (n_params, …
open · @Pix13 · 0 comentários · No GitHub
Server & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows
Descrição
## Title serve: /v1/models meta carries llama-server's model facts (n_params, size, ftype, ...) from the GGUF headers Issue: none found (searched 2026-10-15: n_params, ftype, /v1/models meta). Related: #1882 (other context keys, same dict; the two merge without overlap). ## Summary llama-server's `/v1/models` gives each model a `meta` with `n_params`, `size`, `ftype`, `n_vocab`, `n_ctx_train` and `n_embd`. Strata gives only `n_ctx`. Proxies and dashboards that read llama-server's model list (for example Olla's Models panel) show the parameter count, the quantisation and the size on disk from these keys, and leave them blank for a Strata endpoint. This change reads them from the `--native` GGUF's headers (`tools/gguf_reader.py`, no tensor data), over every shard of a split model, the first time `/v1/models` is asked for, and keeps them: - `n_params`: the elements of every tensor in every shard - `size`: the shards' bytes on disk - `ftype`: llama.cpp's `llama_model_ftype_name()` label of `general.file_type` (`"Q8_0"`, `"IQ3_S - 3.4375 bpw"`) - `n_vocab`, `n_ctx_train`, `n_embd`: `tokenizer.ggml.tokens`, `<arch>.context_length`, `<arch>.embedding_length` Only the facts found are given. A missing shard gives no `n_params`/`size` (rather than a wrong one); an engine without a GGUF (pack only, the mock engine) or a GGUF that cannot be read gives the old `meta`, and an unreadable file is said once on stderr. `meta.n_ctx` is unchanged and stays the engine's. What it leaves alone: the engine, every other endpoint, requests. On opt-in: the change only adds keys to a dict whose existing keys keep their values; llama-server always sends them, so I did not put them behind a setting. Happy to add one if you prefer. ## Measured Header read, on Rocky 9.8 (files in the page cache): | model | shards | read | n_params | size | ftype | |---|---|---|---|---|---| | huihui Qwen3.8-Flash-Next Q8_0 | 6 (188 GB) | 0.23 s | 176943899520 | 188225033440 | `Q8_0` | | ISTA-DASLab GSQ-RCO IQ3_S | 2 (84 GB) | 0.74 s | 176943899520 | 83617662656 | `IQ3_S - 3.4375 bpw` | Both run with this change since 2026-10-15 (2x RTX 3090 and a V100); Olla shows `177B` / `Q8_0` / `175.30 GB`. ## Test - `serve/test_server.py`, new `ModelMeta`: a two-shard GGUF written by the test (named by either shard), a missing shard, no GGUF, an unreadable GGUF (said once), and `/v1/models` with an alias. - `python -m unittest discover -s serve -p 'test_*.py' -t .`: 789 tests, 2 failures that fail on `main` too without this change (`test_keepawake ... early_close`, `test_responses.NoLeakedSampler`, timing-dependent here). - Not tested: Windows, macOS/Metal, a non-split single-file GGUF on a real engine (covered only by the unit test).
No site
Links install, modelos, releases.