Diagnose Distributed AI Workload Failures with NVIDIA Attribution
According to NVIDIA Developer. NVIDIA open-sourced NVIDIA Attribution to correlate workload-scoped logs, metrics, telemetry, and scheduler state into reproducible diagnoses.
- • NVIDIA Attribution is an open-source, workload-centric failure-attribution system used across NVIDIA AI infrastructure.
- • The software consists of two deployable components: the Attribution Service and the Attribution Blueprint reference orchestrator.
- • The attribution workflow scopes workloads, collects observations, extracts and correlates events using deterministic rules, and returns a structured result.
Overview and Architecture
NVIDIA Attribution correlates workload-scoped logs, metrics, telemetry, and scheduler state to produce a structured diagnosis with outcome, root cause, affected nodes, supporting symptoms, and an evidence trail. The system operates through two deployable components: the Attribution Service, which exposes a REST API for the core attribution engine, and the Attribution Blueprint, a Python reference orchestrator that connects cluster schedulers and telemetry sources to the Service.
Four-Stage Process
The attribution path has four stages: scope the workload, collect observations, extract and correlate events through deterministic rules, and return a structured result with terminal state and attributed domain. The Blueprint ships with execution-context connectors for Slurm, Kueue, and static inputs, plus log and metric connectors for Loki, local files, Kubernetes pod logs, and PromQL metrics.
- 1. developer.nvidia.comOpen source