#6641 [Engine] inkling2 Metal: let M1/M2 GPUs compile the S3_K/S5_K gate-up kernels
open · @jiamo · 0 评论 · 去 GitHub 看
Server & APIAMD / HIPNVIDIA / CUDAModels & quants
说明
## Problem and result
On M1 and M2 Macs (Apple7 / Apple8 GPU families), `strata-inkling` 0.1.65-macos-arm64-metal exits with code 3 before it is ready, for every Inkling-2-Mini size:
```
mtl: pipeline ink_moe_gu_situ: Compilation failed due to an interrupted connection: XPC_ERROR_CONNECTION_INTERRUPTED. This error occurred after multiple retries.
```
MTLCompilerService aborts with `cannot select: %5002:gpr(s64) = 66 %5001:gpr, %11878:gpr`. #6608 reports it on an M1 Max and an M1 Pro. I reproduced it on an M2 Max with the released engine and SQ-20 (four crash reports, same reason).
With this change, every kernel in `ink_metal_codecs.metal` compiles on the M2 Max, and SQ-20 serves and answers correctly.
Fixes #6608
## Related discussion
#6608. A focused fix, no scope change.
## Approach
I built a pipeline for each kernel in `ink_metal_codecs.metal` separately, compiled the way `mtl_rt.mm` compiles it (`MTLMathModeSafe`, `supportIndirectCommandBuffers = YES`). On the M2 Max, 4 of 51 fail: `ink_moe_gu_situ`, `ink_moe_gu_sk23`, `ink_moe_gu_sk3`, `ink_moe_gu_sk5`. These are exactly the gate-up kernels that instantiate `sk_w13_chunk<3>` or `sk_w13_chunk<5>`. The 2-, 4-, 6- and 8-bit gate-up kernels compile.
`sk_w8<3>` and `sk_w8<5>` read the high bits with `pl[64 + (j >> 3)]` and `pl[128 + (j >> 3)]`. The fix forms the pointer first and then indexes it (`hp = pl + 64; hp[j >> 3]`). It reads the same byte with the same arithmetic. Changing only these two lines fixes all four kernels, so the shuffle / tile-metadata path suggested in #6608 is not the trigger. Computing the offset in 32 bits instead (`64u + (uint(j) >> 3)`) still crashes the compiler.
The IR in the crash (an exact shift by 3, a zero-extend to s64, then an op with `i64 64`) fits `64 + (j >> 3)`. I matched that by the constants only; the opcode numbers are not documented.
Metal only. No CUDA, HIP or SYCL file is touched.
## Validation
Revision c6f6427d9 plus this commit. Apple M2 Max (Apple8), 96 GB, macOS 27.2.1.
**Passed:**
- Pipeline build of every kernel in `ink_metal_codecs.metal` (probe source below): 4/51 failed before, 0/51 after. `ink_metal_ops.metal` (32), `ink_metal_dspark.metal` (35), `ink_metal_vision.metal` (8) and `ink_metal_mpp.metal` (41) are unchanged and build with 0 failures.
- Bitwise check on the GPU: the patched `sk_w8<3>` / `sk_w8<5>` against the original lines, with `j` formed the way `sk_w13_chunk` forms it, over 65,536 blocks (random bytes, plus 256 blocks that each repeat one byte value). 16,777,216 values per codec, 0 mismatches.
- `cmake -S engine/src/inkling2 -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build strata-inkling` builds. `--version` prints `strata-inkling 0.1.65`, and the binary has the same option strings as the released 0.1.65-macos-arm64-metal engine.
- End to end: that build with Inkling-2-Mini-SQ-20 + Vision, `--max-context 32768`, DSpark on, behind `serve/server.py`. The server reaches ready. At temperature 0, `/v1/chat/completions` answered "What is the capital of France? Answer in one sentence." with "The capital of France is Paris.", and answered a one-sentence question about the Great Wall in Chinese correctly.
- Before the change, on the same machine: the released engine exits with code 3 at `ink_moe_gu_situ`, with the error above.
**Failed:** none observed.
**Not run:**
- SQ-15 and SQ-10 end to end, the sizes in #6608. SQ-20's experts are SF4/MX8, so it may never dispatch the S3_K/S5_K kernels. For those kernels the evidence is the compile and the bitwise check above.
- M1-family GPUs (Apple7): no hardware here. The crash reason in #6608 is identical.
- M3/M4/M5 (Apple9/Apple10): no hardware, so I have not checked that the change is neutral for speed there.
- Pipelines with non-default function-constant values: the probe builds each function with default constants only.
## Limits
This works around an Apple compiler bug rather than fixing it. Other kernels that form a 64-bit index the same way could hit it, but none in these five files fail on the M2 Max with default constants. Affected Macs need a new `strata-inkling` Metal component build to get the fix.
<details><summary>The pipeline probe (Objective-C++)</summary>
Build: `clang++ -std=c++17 -fobjc-arc probe.mm -framework Metal -framework Foundation -o probe`. Run: `./probe engine/src/inkling2/metal/ink_metal_codecs.metal` (all kernels), or name kernels after the file.
```objc
// probe: compile a .metal file like strata's mtl_rt (safe math) and build a pipeline for each kernel.
#import <Metal/Metal.h>
#import <Foundation/Foundation.h>
#include <cstdio>
#include <chrono>
int main(int argc, char** argv) {
@autoreleasepool {
if (argc < 2) { fprintf(stderr, "usage: probe file.metal [kernel ...]\n"); return 2; }
id<MTLDevice> dev = MTLCreateSystemDefaultDevice();
printf("device: %s apple8=%d apple9=%d apple10=%d\n", dev.name.UTF8String,
[dev supportsFamily:MTLGPUFamilyApple8], [dev supportsFamily:MTLGPUFamilyApple9], [dev supportsFamily:(MTLGPUFamily)1010]);
NSError* err = nil;
NSString* src = [NSString stringWithContentsOfFile:@(argv[1]) encoding:NSUTF8StringEncoding error:&err];
if (!src) { fprintf(stderr, "read failed\n"); return 2; }
MTLCompileOptions* o = [MTLCompileOptions new];
o.mathMode = MTLMathModeSafe;
id<MTLLibrary> lib = [dev newLibraryWithSource:src options:o error:&err];
if (!lib) { fprintf(stderr, "library compile failed: %s\n", err.localizedDescription.UTF8String); return 1; }
NSMutableArray<NSString*>* names = [NSMutableArray array];
if (argc > 2) { for (int i = 2; i < argc; ++i) [names addObject:@(argv[i])]; }
else [names addObjectsFromArray:[lib.functionNames sortedArrayUsingSelector:@selector(compare:)]];
int bad = 0;
for (NSString* n in names) {
auto t0 = std::chrono::steady_clock::now();
id<MTLFunction> f = [lib newFunctionWithName:n];
if (f && f.functionConstantsDictionary.count > 0)
f = [lib newFunctionWithName:n constantValues:[MTLFunctionConstantValues new] error:&err];
if (!f) { printf("NOFUNC %s\n", n.UTF8String); ++bad; continue; }
MTLComputePipelineDescriptor* d = [MTLComputePipelineDescriptor new];
d.computeFunction = f; d.supportIndirectCommandBuffers = YES;
err = nil;
id<MTLComputePipelineState> ps = [dev newComputePipelineStateWithDescriptor:d options:MTLPipelineOptionNone reflection:nil error:&err];
double ms = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count();
if (!ps) { printf("FAIL %s (%.0f ms): %s\n", n.UTF8String, ms, err.localizedDescription.UTF8String); ++bad; }
else if (argc > 2) printf("ok %s (%.0f ms)\n", n.UTF8String, ms);
}
printf("%lu kernels, %d failed\n", (unsigned long)names.count, bad);
return bad ? 1 : 0;
}
}
```
</details>
本站相关内容
相关页面的快捷入口。