Hugging Face Blog Publishes Guide to Profiling TorchNeuron on AWS Trainium

TorchNeuron is described as AWS Trainium's open-source native PyTorch backend. AWS Trainium is a purpose-built AI chip for deep-learning training and inference workloads. The post says TorchNeuron supports standard PyTorch workflows on Trainium, including eager execution, torch.compile and standard PyTorch APIs, with minimal code changes. It also integrates with the Neuron Kernel Interface, or NKI, to enable custom operators and performance optimization for Trainium hardware.

The guide says the Neuron backend uses PyTorch's PrivateUse1 extension point, which lets an accelerator integrate with the framework without adding a dedicated dispatch key such as cuda. The torch_neuronx package works with upstream PyTorch to register Neuron as a native backend for PyTorch code, according to the post.

For profiling, the tutorial uses two objects: NeuronConfig and NeuronProfiler. NeuronConfig controls which Neuron data streams and NeuronCores are captured and where output is written; it is passed to torch.profiler.profile as experimental_config. Its modes parameter selects data streams such as device instructions and runtime traces, capture_enabled_for_nc selects the NeuronCores to trace, and correlate_device_time propagates device time into the key_averages() table so CPU and Neuron device time can be compared. NeuronProfiler takes a NeuronConfig and provides an export_trace callback that writes the framework trace, trace.json, alongside Neuron system traces.

The post introduces Neuron Explorer as a browser-based viewer for the system trace, device timeline and per-engine breakdown, accessible through an SSH tunnel or the Visual Studio Code extension. It says the tool can confirm what key_averages() reports numerically and show what happens between dispatches.

After profiling, the output directory contains the system trace and, when device data is captured, matched .neff and .ntff files. The Neuron Executable File Format, or NEFF, contains the compiled Neuron executable, while the matching NTFF contains its hardware instruction trace; the files share a numeric hash that identifies them as a pair. For a system or combined profile, users upload the complete output directory to Neuron Explorer; for a device-only profile, they upload the matching .neff and .ntff files individually.

The example analysis profiles RMSNorm, described as a small operation with a clear sequence of arithmetic and reduction operations. It compares three implementations: naive RMSNorm with individual PyTorch operations including multiply, mean, add, reciprocal square root and multiply; torch.rms_norm(), the PyTorch built-in lowered to a single fused Neuron kernel; and naive RMSNorm with torch.compile, using the same operations as the first implementation. The scripts use the same inputs, profiler configuration and schedule so the results are directly comparable, and the post says all scripts to reproduce the work are available in the torch-neuronx examples directory.

The walkthrough begins with the naive RMSNorm written as individual PyTorch operations. Each line maps to one or more ATen operations and can create intermediate tensors, the post says, before the guide moves on to profiler configuration and trace capture.