Run AI Models for Edge Application Development with TensorRT Edge-LLM
According to NVIDIA Developer. NVIDIA details new developer tools and workflows for prototyping, fine-tuning, and testing AI models and agents on embedded devices using the NVIDIA DGX Spark 64GB.
- • TensorRT Edge-LLM delivers a lightweight C++ inference runtime spanning development and deployment.
- • The trtedge CLI downloads models and builds optimized TensorRT engines in a few commands.
- • DSpark speculative decoding increases mean generation speed during model inference testing.
Edge Application Development Platform
NVIDIA DGX Spark, now available in a 64GB configuration, gives embedded developers a platform to prototype, fine-tune, and test models and agents. NVIDIA TensorRT Edge-LLM provides a lightweight C++ inference runtime designed for embedded workloads, bringing the same inference framework to development on Spark and deployment on platforms such as NVIDIA Jetson Thor, NVIDIA DRIVE AGX Thor, and NVIDIA IGX Thor.
Getting Started with trtedge CLI
You can get started in just a few commands with the trtedge command-line interface. It downloads your selected model and builds TensorRT engines optimized for inference on Spark. Developers can run NVIDIA Nemotron-3.5 Lightning, test prompts in the terminal, and serve the model through a local OpenAI-compatible API on Spark for application prototyping.
Performance and Speculative Decoding
You can speed up generation with DSpark speculative decoding, which uses a draft model to propose several tokens for the main model to verify in a single pass. DSpark speculative decoding with three draft tokens increases mean generation speed from 73.4 to 126.9 tokens per second on HumanEval prompts while peak total system memory use during inference remains 27.9GB.
- 1. developer.nvidia.comOpen source