Issues / #1944

#1944 Long story creation leads to single token responses when context grows

open · @RobinMeyer · 2 comentários · No GitHub

Models & quants

Descrição

I used a story generator with strata 0.1.42, which sends a request for every section with the content of all earlier request as context and there it happened, that after about 12000 tokens, the responses are cut in the middle and all following responses return only one token anymore.
I'm using RTX5070ti 16GB, 7950X3d and 64GB RAM. The configuration is:
"--expert-cache",
  "auto",
  "--prefill",
  "auto",
  "--spec",
  "4",
  "--mtp",
  "D:\\AI\\Strata-data\\mtp\\rt",
  "--max-context",
  "131072",
  "--kv",
  "int8",
  "--kv-resident",
  "32768",
  "--spec-min-p",
  "0.5",
  "--pool-workers",
  "8",
  "--adapt-every",
  "1",
  "--adapt-swaps",
  "160",
  "--adapt-decay",
  "0.97"
With strata 0.1.39, the error seems not to happen. There, I just wrote a 90000 character story without any broken chapters with the same config.

No site

Links install, modelos, releases.