#6642 [Engine] inkling2 GGUF loader: read the merged mainline (#32694) layout

open · @S-Litvinov · 0 comments · View on GitHub

Server & APINVIDIA / CUDAModels & quants

Description

[Engine] inkling2 GGUF loader: read the merged mainline (#32694) layout

## Problem and result

strata-inkling cannot open any Inkling GGUF converted with the merged mainline llama.cpp
converter (ggml-org/Inkling-2-Mini-GGUF and everything built on it, e.g. nokeepai's
Unbound-v2 or AtomicChat): `open: GGUF: missing tensor loop_res_query.weight`.
After this change the same engine loads those files and generates from them.

## Cause

The GGUF loader knows two layouts: the pre-merge PR #32187 "folded" layout and the
Unsloth GGUF_shard_rewrite layout. The merged #32694 layout follows the rewrite naming
for every dense tensor and keeps gate_up experts fused, but it stores three things
differently:

1. there is no global `loop_res_query.weight` / `loop_kv_res_query.weight`; the cross-pass
   queries live per core block as `blk.N.loop_msg_query.weight` [D, 4], each block's four
   rows stacking `[q[2i], kv[2i], kv[2i+1], q[2i+1]]`
   (see llama.cpp `conversion/inkling2.py`, `Inkling2LoopModel.modify_tensors`);
2. the loop controllers are named `loop_{q,k,v,ffn}_ctrl_{down,gain,forget}.weight`
   instead of `loop_{attn,k,v,mlp}_{down,gain,forget}.weight`;
3. `block_count` is the PHYSICAL layer count (24, the loop is described by the separate
   `loop.start/end/count` keys) and `attention.sliding_window_pattern` is indexed per
   physical layer - the pre-merge files store the visit count (48) instead, so the loader
   derived `n_phys` and the SWA pattern wrongly for #32694 files (tail blocks silently
   became non-SWA, e.g. `missing tensor blk.18.attn_q_inp_norm.weight`).

## Approach

One new flag, `is_merged_layout = is_rewrite_layout && no global loop_res_query.weight`,
guarded at exactly the points above: reassemble the canonical [2*core_len, D] q/kv
cross-pass tensors from the per-block tensors, read the controllers under their q/ffn
names, and take `n_phys`/the pattern index straight from the physical layer space.
Metadata reads get `loop.count/start/end` as fallback names next to the old keys.
Every pre-merge path stays byte-identical (the flag is false whenever a global
`loop_res_query.weight` is present or the experts are not fused).

## Validation

Passed:
- `--selfcheck --cpu` PASS (avx2, 12 threads) and `--selfcheck` CUDA PASS on a
  `-DCMAKE_CUDA_ARCHITECTURES=80` build (RTX 3090) with the patch.
- `--info` on `nokeepai/Inkling-2-Mini-Unbound-v2-Q4_K_M.gguf` (#32694 layout, the file
  that failed before): now `arch=inkling2_loop build=GGUF-Q4_K layers=24 visits=48
  drafter=dspark`.
- no regression: `--info` on Unsloth `UD-Q4_K_XL` (rewrite) and on
  `Inkling-2-Mini-SQ-15.strata` (native) prints the same lines as the 0.1.65 release.
- end-to-end generation through the new mapping: `--native <the #32694 file> --gen 24
  --temp 0 --prompt-ids ...` (CPU backend, 42 s) produces fully coherent, factual output
  ("\n\n  1. Reduced memory footprint: By reusing the same transformer weights across
  multiple time steps"). The same file and prompt in llama.cpp b12633 (CUDA) share the
  greedy prefix through "1. Reduced" and diverges at the first content-word choice
  ("latency") - expected cross-kernel greedy sensitivity on 4-bit weights; a wrong
  cross-pass/controller mapping shows up as incoherent tokens from position 1, not as a
  plausible synonym.

Failed / not run:
- GPU (`--serve`/direct) generation with a GGUF `--native` is not reachable in this
  engine version at all: `InkCudaBackend::init` re-opens the `--native` path as
  `.strata`, so ANY community GGUF (also Unsloth UD on the unmodified 0.1.65 release)
  exits with `ERR init: Invalid .strata magic`. Reported separately; out of scope here.
- snapshot tests not run (single-GPU host); the patch touches only the GGUF open path.

## Limits

- `loop.varying` (varloop) is not read; `loop.count` is used as a fixed pass count,
  same as before.
- Verified on Inkling-2-Mini shapes only (core_start 6 / core_end 18 / gate_rank 64).

Related on this site

Quick links to related pages.