#6405 [Engine] 0.2.17: on a 3-card layer split the expert cache shrinks from 5,940 to 3,551 slots (10.40 → 6.19 GiB) at the same VRAM footprint
open · @wulin198550 · 0 评论 · 去 GitHub 看
Setup & installMulti-GPUNVIDIA / CUDA
说明
<html> <body> <!--StartFragment--><h2 style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; font: 700 19px / 28px -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif; margin: 32px 0px 16px; color: rgb(15, 17, 21); letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Problem</h2><p style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; margin: 16px 0px; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Upgrading 0.2.16 (engine 0.1.62) → 0.2.17 (engine 0.1.63) on the same machine, model and config costs 4.21 GiB of expert cache:<span> </span><strong style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; font-weight: 600;">5,940 slots (10.40 GiB) → 3,551 slots (6.19 GiB)</strong>. The VRAM the cards use is the same in both versions (within 2 MiB per card), so the space did not go to another buffer that the log names; the cache simply gets less.</p><p style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; margin: 16px 0px; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">The measured expert-cache hit rate falls with it, and prefill on long prompts is the case that suffers.</p><div class="_tableScroll_13jw0_199 _tableFill_13jw0_252" style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; max-width: 100%; overflow-x: auto; overscroll-behavior-x: contain; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> Version | Expert cache | VRAM used, GPU0 / GPU1 / GPU2 -- | -- | -- 0.2.16 (engine 0.1.62) | 5,940 slots, 10.40 GiB | 15,329 / 15,309 / 23,681 MiB 0.2.17 (engine 0.1.63) | 3,551 slots, 6.19 GiB | 15,331 / 15,311 / 23,683 MiB </div><p style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; margin: 16px 0px; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">All five runs are byte-identical in VRAM (15,331 / 15,311 / 23,683 MiB) and in slot count, and the engine log's<span> </span><code style="corner-shape: superellipse(1.5); text-autospace: no-autospace; --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; display: inline-flex; align-items: center; box-sizing: border-box; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; line-height: 19px; font-optical-sizing: auto; font-size-adjust: none; font-kerning: auto; font-feature-settings: normal; font-variation-settings: normal; font-language-override: normal; font-family: "SF Mono", "JetBrains Mono", "Fira Code", Consolas, "Liberation Mono", Menlo, Courier, "PingFang SC", "Microsoft YaHei", "SF Mono", "JetBrains Mono", "Fira Code", Consolas, "Liberation Mono", Menlo, Courier, "PingFang SC", "Microsoft YaHei"; background-color: rgb(250, 250, 250); border: 0.666667px solid rgba(0, 0, 0, 0.04); border-radius: 8px; padding: 0px 5px; font-size: 0.875em !important;">engine started</code><span> </span>timestamp changes on each run (verified), so the config is being read.</p><p style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; margin: 16px 0px; color: rgb(15, 17, 21); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Things that did<span> </span><strong style="corner-shape: superellipse(1.5); --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; font-weight: 600;">not</strong><span> </span>fix it and are probably not related:<span> </span><code style="corner-shape: superellipse(1.5); text-autospace: no-autospace; --dsw-elevation-stroke: 0 0 0 .5px #00000029; --dsw-elevation-panel: 0 0 0 .5px #00000029, 0 3px 8px 0 #00000008, 0 0 16px 0 #00000005; --dsw-elevation-prominent: 0 0 0 .5px #00000029, 0 3px 8px 0 #0000000a, 0 0 20px 0 #0000000d; --dsw-elevation-soft: 0 0 0 .5px #00000029, 0 4px 16px 0 #00000008, 0 0 24px 0 #00000008; display: inline-flex; align-items: center; box-sizing: border-box; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; line-height: 19px; font-optical-sizing: auto; font-size-adjust: none; font-kerning: auto; font-feature-settings: normal; font-variation-settings: normal; font-language-override: normal; font-family: "SF Mono", "JetBrains Mono", "Fira Code", Consolas, "Liberation Mono", Menlo, Courier, "PingFang SC", "Microsoft YaHei", "SF Mono", "JetBrains Mono", "Fira Code", Consolas, "Liberation Mono", Menlo, Courier, "PingFang SC", "Microsoft YaHei"; background-color: rgb(250, 250, 250); border: 0.666667px solid rgba(0, 0, 0, 0.04); border-radius: 8px; padding: 0px 5px; font-size: 0.875em !important;">--prefill auto:65536</code>,<span> </span><code style="corner-shape: superellipse(1.5); text-autospace: no-autospace; --dsw-eleva
本站相关内容
相关页面的快捷入口。