问题 / #2820

#2820 [Feature Request]: add missing llama.cpp server endpoints (/tokenize, /completion, /embedding, /metrics)

open · @jeremyoha450 · 0 评论 · 去 GitHub 看

Server & API

说明

Posible to add more of the llama outputs, etc? (i've add some myself. but other might like them too)

Token and template routes

    POST /tokenize is missing upstream. A local patch adds it, plus /v1/tokenize, with with_pieces.
    POST /detokenize is missing upstream. A local patch adds it, plus /v1/detokenize.
    POST /apply-template is missing upstream. A local patch adds it, plus /v1/apply-template, including response_format schema text and input_tokens.
    There is no llama.cpp-compatible add_special / BOS handling on tokenize.
    There is no /tokenize of a chat message list without a separate apply-template call.

Native generation

    POST /completion is missing, including the native stream fields content, stop, stop_type, tokens_predicted, tokens_evaluated, tokens_cached, and generation_settings.
    POST /completions is missing.
    POST /infill is missing.
    POST /v1/completions is missing (legacy non-chat completions).

Embeddings and adapters

    POST /embedding is missing.
    POST /embeddings is missing.
    POST /v1/embeddings is missing.
    GET /lora-adapters is missing.
    POST /lora-adapters is missing.

Slots and metrics

    POST /slots/{id}?action=erase is missing. Save and restore exist.
    Prometheus text on GET /metrics is missing. Strata’s /metrics is a different JSON document.
    The llama.cpp timings object is missing from chat chunks: prompt_ms, predicted_ms, prompt_per_token_ms, predicted_per_token_ms, prompt_per_second, predicted_per_second.

本站相关内容

相关页面的快捷入口。