Issues / #2820
#2820 [Feature Request]: add missing llama.cpp server endpoints (/tokenize, /completion, /embedding, /metrics)
open · @jeremyoha450 · 0 Kommentare · Auf GitHub
Beschreibung
Posible to add more of the llama outputs, etc? (i've add some myself. but other might like them too)
Token and template routes
POST /tokenize is missing upstream. A local patch adds it, plus /v1/tokenize, with with_pieces.
POST /detokenize is missing upstream. A local patch adds it, plus /v1/detokenize.
POST /apply-template is missing upstream. A local patch adds it, plus /v1/apply-template, including response_format schema text and input_tokens.
There is no llama.cpp-compatible add_special / BOS handling on tokenize.
There is no /tokenize of a chat message list without a separate apply-template call.
Native generation
POST /completion is missing, including the native stream fields content, stop, stop_type, tokens_predicted, tokens_evaluated, tokens_cached, and generation_settings.
POST /completions is missing.
POST /infill is missing.
POST /v1/completions is missing (legacy non-chat completions).
Embeddings and adapters
POST /embedding is missing.
POST /embeddings is missing.
POST /v1/embeddings is missing.
GET /lora-adapters is missing.
POST /lora-adapters is missing.
Slots and metrics
POST /slots/{id}?action=erase is missing. Save and restore exist.
Prometheus text on GET /metrics is missing. Strata’s /metrics is a different JSON document.
The llama.cpp timings object is missing from chat chunks: prompt_ms, predicted_ms, prompt_per_token_ms, predicted_per_token_ms, prompt_per_second, predicted_per_second.
Mehr auf der Site
Links zu Install, Modellen, Releases.