3 Ways to Boost MoE Inference on NVIDIA DGX Spark
According to NVIDIA Developer. Learn how developer techniques and kernel replacements optimize Mixture-of-Experts inference performance for the Ling-3.0-flash model on NVIDIA DGX Spark.
- • Replacing the FlashInfer MXFP4 MoE backend with the Humming kernel library yielded a 9.3% throughput gain.
- • Online FP8 quantization of the LM head halved its weight-read volume and improved decode throughput.
- • Multi-token prediction speculative decoding using the NEXTN layer achieved 3.90 accepted tokens per round on a synthetic workload.
- • Combining all three optimizations increased decode throughput by 42.5% and reduced mean latency by 24.5% on the GSM8K test set.
Evaluation Overview
Ling-3.0-flash is a 124-billion-parameter mixture-of-experts model with 5.1 billion active parameters per token, evaluated on NVIDIA DGX Spark at batch size 1. Profiling showed that decoding was memory-bound, with the mixture-of-experts computation and output-layer projection driving execution time.
Humming Backend and FP8 Quantization
Replacing the FlashInfer MXFP4 MoE backend with the Humming kernel library reduced time per output token from 27.71 ms to 25.35 ms, a 9.3% throughput gain. Online FP8 quantization of the LM head halved its weight-read volume, which, combined with Humming, improved decode throughput by 21.7% over the baseline.
Speculative Decoding and Combined Results
Multi-token prediction speculative decoding using the model's built-in NEXTN layer achieved 3.90 accepted tokens per round on a synthetic workload, raising throughput 185.1% over the baseline. On the full GSM8K test set, the three optimizations together increased decode throughput by 42.5% and reduced mean latency by 24.5% with no statistically detectable accuracy change.
- 1. developer.nvidia.comOpen source