#5830 [Engine] Unpack IQ4_NL PLE rows with AVX1
closed · draft · @aktalascloud · 0 comentarios · En GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Descripción
## Problem and result
IQ4_NL PLE rows are unpacked scalar on the AVX-only build. On a Xeon E5-2697 v2, unpacking one 160-value row fell from 157.08 to 53.53 ns (six runs per arm).
## Related discussion
None.
## Approach
Use two 128-bit byte shuffles per block and keep the existing split-half order and FP16 scaling. Compile `ngram.cpp` with AVX flags in the CUDA/HIP `STRATA_ISA_FLOOR=avx` build so the code is active in the normal build target.
## Validation
**Passed**
- Built `strata` at `cbeeec00` on Windows 10 (19044), MSVC 19.44, CUDA 12.9, and Tesla P40 (sm_61). Configure Release with Ninja, `-DSTRATA_ENABLE_CUDA=ON -DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_ISA_FLOOR=avx -DCMAKE_CUDA_ARCHITECTURES=61 -DSTRATA_BUILD_TESTS=OFF -DGGML_CCACHE=OFF` and `-DCMAKE_CUDA_FLAGS="--allow-unsupported-compiler -Xcompiler=/arch:AVX"`. Run `cmake --build build --target strata --parallel 10`, then `build/strata.exe --version` (0.1.59.1). The CMake `ngram.cpp` object contains the two byte shuffles.
- The row check matched all 65,536 FP16 encodings across 160 outputs each, bit for bit. Default and `/arch:AVX2` row disassembly matched `5f0ae5b5` after normalizing the object-specific anonymous namespace name.
- Microbenchmark pinned to logical processor 3 on the same dual Xeon host. Each run decoded 32,768 deterministic rows 80 times. Checksums matched in all runs.
| Variant | Median ns/row | Individual runs (ns/row) |
| --- | ---: | --- |
| `5f0ae5b5`, default flags | 157.08 | 157.566, 156.557, 157.184, 156.968, 156.937, 157.261 |
| `5f0ae5b5`, `/arch:AVX` | 159.77 | 159.914, 159.618, 160.345, 159.303, 159.575, 160.310 |
| `cbeeec00`, CMake AVX object | 53.53 | 53.635, 53.128, 53.713, 53.259, 53.503, 53.562 |
**Failed**
- The initial standalone harness link omitted the PLE reader and platform objects (`LNK2019`). Linking those objects resolved it.
**Not run**
- HIP and SYCL builds, artifact-backed `ple_parity --selftest`, and a matched full-model A/B run. An AVX-floor HIP compile and a SYCL compile would help complete backend coverage.
## Limits
The speedup measures host row unpacking on Ivy Bridge. Full-request speed remains unmeasured for this revision.
<details>
<summary>Standalone row check and reproduction</summary>
Save the source below as `ple-row-check.cpp`. In the x64 MSVC developer shell, compile it with `cl /nologo /O2 /Ob2 /MD /EHsc /std:c++20 /arch:AVX /Iinclude /c ple-row-check.cpp`. Compile the baseline `src/kernels/ngram.cpp` with the same flags, first omitting and then adding `/arch:AVX`. For the candidate, use the CMake object at `build/CMakeFiles/strata_kernels.dir/src/kernels/ngram.cpp.obj`.
Link each row object and `ple-row-check.obj` with `cl /nologo /MD <row.obj> ple-row-check.obj build/CMakeFiles/strata_kernels.dir/src/ngram/ple_reader.cpp.obj build/CMakeFiles/strata_kernels.dir/src/platform/direct_file.cpp.obj build/strata_core.lib dxgi.lib /link /OPT:REF /OUT:ple-row-check.exe`. Run `ple-row-check.exe` for parity and `ple-row-check.exe --bench` for timing.
```cpp
#include "strata/kernels/f16_bits.hpp"
#include "strata/kernels/ngram.hpp"
#include <array>
#include <bit>
#include <chrono>
#include <cstdio>
#include <cstring>
#include <string_view>
#include <vector>
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
using namespace strata::kernels;
using Row = std::array<uint8_t, PLE_ROW_BYTES>;
using Values = std::array<float, PLE_HEAD_DIM>;
void fill_row(Row& row, unsigned seed, bool exhaustive) {
for (int b = 0; b < PLE_HEAD_DIM / 32; ++b) {
uint8_t* block = row.data() + 18 * b;
const unsigned bits = seed + 0x193d * b;
const uint16_t half = exhaustive ? uint16_t(bits) : uint16_t(0x2400 | (bits & 1023));
std::memcpy(block, &half, sizeof(half));
for (unsigned j = 0; j < 16; ++j) {
const unsigned low = (j + seed) & 15;
const unsigned high = (15 - j + (seed >> 4)) & 15;
block[2 + j] = uint8_t((high << 4) | low);
}
}
}
int check() {
Row row{};
Values output{};
for (unsigned pattern = 0; pattern < 65536; ++pattern) {
fill_row(row, pattern, true);
iq4nl_dequant_row(row.data(), output.data());
for (int b = 0; b < PLE_HEAD_DIM / 32; ++b) {
const uint8_t* block = row.data() + 18 * b;
uint16_t half;
std::memcpy(&half, block, sizeof(half));
const float scale = f32_from_f16(half);
for (int j = 0; j < 32; ++j) {
const int packed = block[2 + j % 16];
const int code = j < 16 ? packed & 15 : packed >> 4;
const float expected = scale * float(iq4nl_code(code));
if (std::bit_cast<uint32_t>(output[32 * b + j]) != std::bit_cast<uint32_t>(expected)) {
std::fprintf(stderr, "mismatch scale %04x, block %d, value %d\n", half, b, j);
return 1;
}
}
}
}
std::puts("65536 half encodings x 160 IQ4_NL values match bitwise.");
return 0;
}
int benchmark() {
if (SetThreadAffinityMask(GetCurrentThread(), DWORD_PTR(1) << 3) == 0) {
std::fputs("could not pin worker to logical processor 3\n", stderr);
return 1;
}
std::vector<Row> rows(32768);
for (unsigned i = 0; i < rows.size(); ++i) {
fill_row(rows[i], i * 0x9e3779b9u, false);
}
Values output{};
uint32_t checksum = 0;
for (const Row& row : rows) {
iq4nl_dequant_row(row.data(), output.data());
}
constexpr int repeat = 80;
const auto start = std::chrono::steady_clock::now();
for (int r = 0; r < repeat; ++r) {
for (unsigned i = 0; i < rows.size(); ++i) {
iq4nl_dequant_row(rows[i].data(), output.data());
checksum += std::bit_cast<uint32_t>(output[(i + r) % PLE_HEAD_DIM]);
}
}
const double ns = std::chrono::duration<double, std::nano>(std::chrono::steady_clock::now() - start).count();
std::printf("%.3f ns/row checksum %08x\n", ns / (rows.size() * repeat), checksum);
return 0;
}
int main(int argc, char** argv) {
return argc > 1 && std::string_view(argv[1]) == "--bench" ? benchmark() : check();
}
```
</details>
En el sitio
Enlaces a install, modelos, releases.