diff --git a/TECHNICAL_REPORTS/1878-fft-numeric-probe-20261015.en.md b/TECHNICAL_REPORTS/1878-fft-numeric-probe-20261015.en.md new file mode 100644 index 000000000..b63c6e3e0 --- /dev/null +++ b/TECHNICAL_REPORTS/1878-fft-numeric-probe-20261015.en.md @@ -0,0 +1,33 @@ +# PR #1878: Reliable FFT numeric and plan-cache probes + +**Date**: 2026-10-15 +**Status**: Implemented and validated on CUDA +**Risk**: Low; standalone diagnostic example only + +## Summary + +Promote the scratch FFT programs from #1825 and #1876 into a reusable mlxcel-core example. The default mode compares real and complex transforms with the CPU stream, while `--plans N` exercises distinct forward/inverse transform shapes under the backend's plan-cache limit. Integrate main `cb73c7d6` and MLX `243bdf09` through the mlxcelverse layout. + +## Problem + +The submitted stress mode printed numerical failures but always returned success, allowing a script to accept incorrect results. Missing or malformed plan counts silently became 40, zero performed no work, and unchecked counts could overflow the i32 transform or batch dimensions. The previous coverage also omitted odd real-transform lengths. + +## Decisions and changes + +Both numerical modes share comparison accounting: non-finite, negative, or out-of-tolerance errors remain failures even after later successful comparisons, and produce exit status 1. A strict parser rejects invalid, zero, extra, and arithmetic-overflow arguments with status 2 before device work. The upper bound comes from the i32 batch-times-length geometry, not a benchmark budget. Help returns success. + +Add lengths 3, 7, and 401, including odd inverse transforms and a complex transform, for 35 comparisons. Preserve the CPU stream reference, evaluation within each device guard, deterministic inputs, and the 1e-5 tolerance. Clarify that shape count is not retained-plan count: forward/inverse plans may be evicted at the configured limit. Document backend features and build before applying a runtime timeout, so cold compilation cannot be mistaken for the historical ROCm hang. CPU-only builds remain degenerate smoke tests. + +## Validation + +On NVIDIA GB10, all 35 CUDA/CPU comparisons pass, with maximum reported relative error 9.538e-7. Forty distinct stress shapes complete with maximum round-trip error 8.946e-7 and no numerical failures. Both modes return 0. Help returns 0; missing, zero, malformed, overflowing, and extra arguments each return 2 in actual subprocess tests. Four unit tests cover parsing boundaries, geometry, and failure persistence including NaN/infinities and the tolerance boundary. Full CUDA workspace/all-targets Clippy with warnings denied, formatting, license-header, and diff checks pass. + +Validated source tree: `590b55dd21d412bd7d071ce6d3ffbd66abd6f673`. Probe SHA-256: `8377fca87228e0d66d87c823735f5a9c1f0d7740745754952062d0ef4f845913`. Reports added afterward do not alter that production source. + +The original PR recorded gfx1151 comparisons below 5e-7 and a 40-shape run at cache size 8, versus a hang near shape 17 at size 32. Those are historical measurements, not fresh results on this pin. AMD hardware is unavailable here; this PR supplies the reproducer and does not claim to fix hipFFT allocation or the cache-size hang. + +## Change summary + +One executable example with embedded unit tests and bilingual reports; no runtime backend, dependency, or cache policy changes. No blocking correctness, security, or performance issue remains in the diagnostic tool. + +Refs #1825, #1876. diff --git a/TECHNICAL_REPORTS/1878-fft-numeric-probe-20261015.ko.md b/TECHNICAL_REPORTS/1878-fft-numeric-probe-20261015.ko.md new file mode 100644 index 000000000..fbf0ddb37 --- /dev/null +++ b/TECHNICAL_REPORTS/1878-fft-numeric-probe-20261015.ko.md @@ -0,0 +1,33 @@ +# PR #1878: FFT 수치 및 plan-cache 검증 도구 + +**작성일**: 2026-10-15 +**상태**: 구현 및 CUDA 검증 완료 +**위험도**: 낮음; 독립 진단 예제만 변경 + +## 요약 + +#1825와 #1876에서 사용한 임시 FFT 프로그램을 재실행 가능한 mlxcel-core 예제로 만든다. 기본 모드는 실수·복소수 변환을 CPU stream과 비교하며, `--plans N`은 backend cache 한도에서 서로 다른 forward/inverse transform shape를 실행한다. Main `cb73c7d6`와 MLX `243bdf09`를 mlxcelverse 경로에 맞게 통합했다. + +## 문제 정의 + +제출된 stress 모드는 수치 실패를 출력해도 항상 성공으로 종료하여 스크립트가 잘못된 결과를 승인할 수 있었다. 누락되거나 잘못된 plan 수는 조용히 40으로 대체됐고, 0은 아무 작업도 하지 않았으며, 큰 수는 i32 transform 또는 batch 차원을 넘칠 수 있었다. 기존 범위에는 홀수 길이 실수 변환도 없었다. + +## 기술적 선택과 변경 + +두 수치 모드는 공통 비교 집계를 사용한다. 비유한값, 음수 또는 허용 오차 이상의 값은 뒤의 비교가 성공해도 실패로 유지되고 종료 코드 1을 만든다. 엄격한 parser는 잘못된 값, 0, 추가 인자, 산술 범위 초과를 device 작업 전에 종료 코드 2로 거부한다. 상한은 benchmark 예산이 아니라 i32 batch와 length의 곱에서 도출했다. 도움말은 성공으로 종료한다. + +길이 3, 7, 401과 홀수 inverse·복소수 변환을 추가하여 35개 비교를 실행한다. CPU stream 기준, 각 device guard 내부의 평가, 결정적 입력, 1e-5 허용 오차를 유지한다. Shape 수와 보유 plan 수를 구분하며 forward/inverse plan은 설정된 cache 한도에 따라 퇴출될 수 있음을 명시했다. Backend feature를 설명하고 빌드 후 runtime timeout을 적용하도록 하여 cold compilation을 과거 ROCm hang으로 오인하지 않게 했다. CPU-only 빌드는 양쪽이 CPU인 smoke test다. + +## 검증 + +NVIDIA GB10에서 CUDA/CPU 비교 35개가 모두 통과했고 보고된 최대 상대 오차는 9.538e-7이었다. 서로 다른 stress shape 40개도 수치 실패 없이 완료됐으며 최대 왕복 오차는 8.946e-7이었다. 두 모드는 종료 코드 0을 반환했다. 실제 subprocess 검증에서 도움말은 0, 누락·0·잘못된 값·범위 초과·추가 인자는 각각 2를 반환했다. 단위 테스트 4개는 parser 경계, 차원 산술, NaN·무한대·허용 오차 경계를 포함한 실패 유지 동작을 검증한다. 경고를 오류로 처리한 전체 CUDA workspace/all-targets Clippy, formatting, license-header, diff 검사도 통과했다. + +검증 소스 tree: `590b55dd21d412bd7d071ce6d3ffbd66abd6f673`. Probe SHA-256: `8377fca87228e0d66d87c823735f5a9c1f0d7740745754952062d0ef4f845913`. 이후 추가한 보고서는 해당 프로덕션 소스를 바꾸지 않는다. + +원래 PR은 gfx1151에서 5e-7 미만의 오차, cache size 8의 40-shape 완료, size 32의 약 17번째 shape hang을 기록했다. 이는 과거 측정이며 현재 pin의 새 결과가 아니다. 여기에는 AMD 하드웨어가 없어 재현 도구만 제공하며 hipFFT 할당이나 cache 크기 관련 hang의 해결을 주장하지 않는다. + +## 변경 요약 + +단위 테스트를 포함한 실행 예제 하나와 영문·국문 보고서를 추가했다. Runtime backend, 의존성, cache 정책은 바꾸지 않는다. 진단 도구에 남은 blocking 정확성·보안·성능 문제는 발견되지 않았다. + +Refs #1825, #1876. diff --git a/src/lib/mlxcel-core/examples/fft_numeric_probe.rs b/src/lib/mlxcel-core/examples/fft_numeric_probe.rs new file mode 100644 index 000000000..3953dbab4 --- /dev/null +++ b/src/lib/mlxcel-core/examples/fft_numeric_probe.rs @@ -0,0 +1,360 @@ +// Copyright 2025-2026 Lablup Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//! GPU-against-CPU check for the device FFT, plus the plan-cache stress that +//! found the hipFFT hang in lablup/mlxcel#1876. +//! +//! Two modes, because the two questions are different and one of them can hang. +//! +//! cargo run --profile test-fast -p mlxcel-core --features rocm --example fft_numeric_probe +//! cargo run --profile test-fast -p mlxcel-core --features rocm --example fft_numeric_probe -- --plans 40 +//! +//! Select `cuda` or `metal` instead for those backends. Without a backend feature, +//! both arms use the CPU; that is a smoke test, not GPU validation. +//! +//! `correctness` (the default) runs `rfft`, `irfft`, a round trip and a complex +//! `fft` on the GPU and on the CPU stream and reports the relative difference. +//! It covers the lengths mlxcel actually asks for (Kokoro's STFT at 1200, +//! Phi-4-multimodal at 512) plus a non-power-of-two, and batches from 3 to 512. +//! +//! `--plans N` is the stress: N distinct transform shapes in one process, +//! exercising plan creation and eviction at the configured cache limit. +//! Each shape runs both a forward and an inverse transform. On ROCm, past roughly a +//! dozen live hipFFT plans the next plan creation never returns, with the GPU +//! idle and the calling thread asleep (lablup/mlxcel#1876). The cap that bounds +//! it is `MLX_ROCM_FFT_CACHE_SIZE`, default 8, so +//! +//! cargo build --profile test-fast -p mlxcel-core --features rocm --example fft_numeric_probe +//! MLX_ROCM_FFT_CACHE_SIZE=32 timeout 120s target/test-fast/examples/fft_numeric_probe --plans 40 +//! +//! is the reproduction, and the same command without the variable is the +//! control. Run it under `timeout` if you are scripting it: the failure is a +//! hang, not an error, so nothing comes back on its own. +//! Build before applying the timeout so compilation is not mistaken for a hang. +//! Numerical failures exit with status 1; invalid arguments exit with status 2. +//! +//! The reference is the CPU stream rather than another host. Two accelerators +//! are not expected to agree bit for bit, and the CPU arm is available wherever +//! this runs, so a difference here is this backend's and not a portability +//! question. See `docs/benchmark_results/rocm-correctness-gfx1151-2026-09-12.md` +//! for why byte-identity is not the cross-backend criterion. + +use std::process::ExitCode; + +use mlxcel_core::streams::DefaultDeviceGuard; +use mlxcel_core::{self, MlxArray, UniquePtr, dtype}; + +/// Transform lengths worth covering: 2 and 8 are degenerate, 400 is Whisper's +/// frame and not a power of two, 512 is the Phi-4-multimodal front end, 1200 is +/// Kokoro's STFT, and 1024/2048 bracket them. Odd lengths exercise the inverse +/// real-transform output size rather than only its even-length default. +const LENGTHS: &[i32] = &[2, 3, 7, 8, 400, 401, 512, 1024, 2048]; +const KOKORO_N_FFT: i32 = 1200; +const PLAN_BATCH: i32 = 4; +const FIRST_PLAN_LENGTH: i32 = 64; +const PLAN_LENGTH_STEP: i32 = 8; +// Bound both the transform dimension and input's i32 batch * length product. +const MAX_PLAN_COUNT: usize = + ((i32::MAX / PLAN_BATCH - FIRST_PLAN_LENGTH) / PLAN_LENGTH_STEP + 1) as usize; +const USAGE: &str = "Usage: fft_numeric_probe [--plans N | --help]"; + +#[derive(Debug, PartialEq, Eq)] +enum Mode { + Correctness, + Plans(usize), + Help, +} + +fn parse_mode(args: &[String]) -> Result { + match args { + [] => Ok(Mode::Correctness), + [flag] if flag == "--help" || flag == "-h" => Ok(Mode::Help), + [flag, value] if flag == "--plans" => { + let count = value + .parse::() + .map_err(|_| format!("invalid plan count: {value}"))?; + if !(1..=MAX_PLAN_COUNT).contains(&count) { + return Err(format!("plan count must be in 1..={MAX_PLAN_COUNT}")); + } + Ok(Mode::Plans(count)) + } + _ => Err("expected no arguments, --plans N, or --help".to_owned()), + } +} + +#[derive(Debug, Default)] +struct Comparisons { + failures: usize, +} + +impl Comparisons { + fn record(&mut self, relative_error: f32) -> bool { + let passed = relative_error.is_finite() && (0.0..TOLERANCE).contains(&relative_error); + self.failures += usize::from(!passed); + passed + } + + fn exit_code(&self) -> ExitCode { + if self.failures == 0 { + ExitCode::SUCCESS + } else { + ExitCode::FAILURE + } + } +} + +fn pseudo_random(count: usize, seed: u32) -> Vec { + let mut state = seed; + (0..count) + .map(|_| { + state = state.wrapping_mul(1_664_525).wrapping_add(1_013_904_223); + ((state >> 8) as f32 / (1u32 << 24) as f32) * 2.0 - 1.0 + }) + .collect() +} + +fn max_abs(a: &MlxArray) -> f32 { + let f = mlxcel_core::astype(a, dtype::FLOAT32); + let m = mlxcel_core::max_all(&mlxcel_core::abs(&f)); + mlxcel_core::eval(&m); + mlxcel_core::item_f32(&m) +} + +fn max_abs_diff(a: &MlxArray, b: &MlxArray) -> f32 { + max_abs(&mlxcel_core::subtract( + &mlxcel_core::astype(a, dtype::FLOAT32), + &mlxcel_core::astype(b, dtype::FLOAT32), + )) +} + +/// Relative difference of two complex arrays, real and imaginary parts summed +/// and scaled by the reference's magnitude. +fn complex_rel(gpu: &MlxArray, cpu: &MlxArray) -> f32 { + let scale = max_abs(&mlxcel_core::real_part(cpu)) + max_abs(&mlxcel_core::imag_part(cpu)); + let d = max_abs_diff(&mlxcel_core::real_part(gpu), &mlxcel_core::real_part(cpu)) + + max_abs_diff(&mlxcel_core::imag_part(gpu), &mlxcel_core::imag_part(cpu)); + d / if scale > 0.0 { scale } else { 1.0 } +} + +fn real_rel(gpu: &MlxArray, cpu: &MlxArray) -> f32 { + let scale = max_abs(cpu); + max_abs_diff(gpu, cpu) / if scale > 0.0 { scale } else { 1.0 } +} + +fn input(batch: i32, n: i32, seed: u32) -> UniquePtr { + let v = pseudo_random((batch * n) as usize, seed); + mlxcel_core::from_slice_f32(&v, &[batch, n]) +} + +/// Run `f` with the default device pinned, evaluating before the guard drops so +/// the work actually lands on that device rather than wherever it resolves +/// later. +fn on_gpu UniquePtr>(f: F) -> UniquePtr { + let _guard = DefaultDeviceGuard::gpu(); + let out = f(); + mlxcel_core::eval(&out); + out +} + +fn on_cpu UniquePtr>(f: F) -> UniquePtr { + let _guard = DefaultDeviceGuard::cpu(); + let out = f(); + mlxcel_core::eval(&out); + out +} + +const TOLERANCE: f32 = 1e-5; + +fn correctness() -> Comparisons { + println!( + "FFT, GPU against the CPU stream. Pass is a relative difference under {TOLERANCE:.0e}.\n" + ); + println!( + "{:<10}{:>7}{:>8}{:>14}{:>8}", + "case", "batch", "n", "rel", "verdict" + ); + + let mut comparisons = Comparisons::default(); + let mut report = |case: &str, batch: i32, n: i32, rel: f32| { + let ok = comparisons.record(rel); + println!( + "{:<10}{:>7}{:>8}{:>14.3e}{:>8}", + case, + batch, + n, + rel, + if ok { "ok" } else { "FAIL" } + ); + }; + + let mut lengths = LENGTHS.to_vec(); + lengths.push(KOKORO_N_FFT); + + for &n in &lengths { + let x = input(4, n, 0xC0FFEE ^ n as u32); + let g = on_gpu(|| mlxcel_core::rfft(&x, n, 1)); + let c = on_cpu(|| mlxcel_core::rfft(&x, n, 1)); + report("rfft", 4, n, complex_rel(&g, &c)); + } + + for &n in &lengths { + let x = input(4, n, 0xBEEF00 ^ n as u32); + // The spectrum comes from the CPU so the inverse is compared on the same + // input rather than on each backend's own forward result. + let spec = on_cpu(|| mlxcel_core::rfft(&x, n, 1)); + let g = on_gpu(|| mlxcel_core::irfft(&spec, n, 1)); + let c = on_cpu(|| mlxcel_core::irfft(&spec, n, 1)); + report("irfft", 4, n, real_rel(&g, &c)); + } + + for &n in &lengths { + let x = input(3, n, 0x1234 ^ n as u32); + let back = on_gpu(|| { + let spec = mlxcel_core::rfft(&x, n, 1); + mlxcel_core::irfft(&spec, n, 1) + }); + report("roundtrip", 3, n, real_rel(&back, &x)); + } + + for &n in &[7, 8, 512, KOKORO_N_FFT] { + let x = mlxcel_core::astype(&input(4, n, 0x77 ^ n as u32), dtype::COMPLEX64); + let g = on_gpu(|| mlxcel_core::fft(&x, n, 1)); + let c = on_cpu(|| mlxcel_core::fft(&x, n, 1)); + report("fft", 4, n, complex_rel(&g, &c)); + } + + // A batch large enough that the launch geometry is not the trivial one. + { + let (batch, n) = (512, 512); + let x = input(batch, n, 0xABCD); + let g = on_gpu(|| mlxcel_core::rfft(&x, n, 1)); + let c = on_cpu(|| mlxcel_core::rfft(&x, n, 1)); + report("rfft", batch, n, complex_rel(&g, &c)); + } + + println!(); + let failures = comparisons.failures; + if failures == 0 { + println!("all FFT results match the CPU stream"); + } else { + println!("{failures} comparison(s) disagreed with the CPU stream"); + } + comparisons +} + +/// Exercise `count` distinct transform shapes in one process. The backend's +/// cache capacity determines how many forward/inverse plans remain live. +fn plans(count: usize) -> Comparisons { + println!( + "Plan-cache stress: {count} distinct shapes in one process.\n\ + On ROCm this hangs rather than failing once too many plans are live \ + (lablup/mlxcel#1876);\n\ + run it under `timeout` and treat no output as the failure.\n" + ); + let mut comparisons = Comparisons::default(); + for i in 0..count { + // The parser bounds count so the dimension and batch product fit i32. + let n = FIRST_PLAN_LENGTH + (i as i32) * PLAN_LENGTH_STEP; + let x = input(PLAN_BATCH, n, 0x5EED ^ n as u32); + let spec = on_gpu(|| mlxcel_core::rfft(&x, n, 1)); + let back = on_gpu(|| mlxcel_core::irfft(&spec, n, 1)); + let rel = real_rel(&back, &x); + let ok = comparisons.record(rel); + println!( + "plan {:>3} n={:<6} roundtrip rel={:.3e} {}", + i, + n, + rel, + if ok { "ok" } else { "FAIL" } + ); + } + println!( + "\ncompleted {count} transform shapes without blocking; {} numerical failure(s)", + comparisons.failures + ); + comparisons +} + +fn main() -> ExitCode { + let args: Vec = std::env::args().skip(1).collect(); + match parse_mode(&args) { + Ok(Mode::Correctness) => correctness().exit_code(), + Ok(Mode::Plans(count)) => plans(count).exit_code(), + Ok(Mode::Help) => { + println!("{USAGE}"); + ExitCode::SUCCESS + } + Err(error) => { + eprintln!("{error}\n{USAGE}"); + ExitCode::from(2) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn parse(args: &[&str]) -> Result { + parse_mode(&args.iter().map(|arg| (*arg).to_owned()).collect::>()) + } + + #[test] + fn accepts_default_help_and_explicit_plan_counts() { + assert_eq!(parse(&[]), Ok(Mode::Correctness)); + assert_eq!(parse(&["--help"]), Ok(Mode::Help)); + assert_eq!(parse(&["-h"]), Ok(Mode::Help)); + assert_eq!(parse(&["--plans", "40"]), Ok(Mode::Plans(40))); + assert_eq!( + parse(&["--plans", &MAX_PLAN_COUNT.to_string()]), + Ok(Mode::Plans(MAX_PLAN_COUNT)) + ); + } + + #[test] + fn rejects_missing_malformed_empty_and_overflowing_plan_requests() { + for args in [ + vec!["--plans"], + vec!["--plans", "no"], + vec!["--plans", "-1"], + vec!["--plans", "0"], + vec!["--plans", "1", "extra"], + vec!["--unknown"], + ] { + assert!(parse(&args).is_err(), "accepted {args:?}"); + } + assert!(parse(&["--plans", &(MAX_PLAN_COUNT + 1).to_string()]).is_err()); + assert!(parse(&["--plans", &usize::MAX.to_string()]).is_err()); + } + + #[test] + fn bounded_plan_geometry_does_not_overflow_the_input_product() { + let last = FIRST_PLAN_LENGTH + (MAX_PLAN_COUNT as i32 - 1) * PLAN_LENGTH_STEP; + assert!(last.checked_mul(PLAN_BATCH).is_some()); + assert!((last + PLAN_LENGTH_STEP).checked_mul(PLAN_BATCH).is_none()); + } + + #[test] + fn numerical_failures_survive_later_passing_comparisons() { + let mut comparisons = Comparisons::default(); + assert!(comparisons.record(0.0)); + assert!(comparisons.record(TOLERANCE / 2.0)); + assert_eq!(comparisons.exit_code(), ExitCode::SUCCESS); + for error in [TOLERANCE, f32::NAN, f32::INFINITY, f32::NEG_INFINITY, -1.0] { + assert!(!comparisons.record(error)); + } + assert!(comparisons.record(0.0)); + assert_eq!(comparisons.failures, 5); + assert_eq!(comparisons.exit_code(), ExitCode::FAILURE); + } +}