Meet Us at SC26
This week, we are at SC26 in Chicago. We present three research posters on tensor contractions for multicore CPUs, AWS Trainium2 and AMD’s XDNA2. The posters are on display from Tuesday to Thursday, November 17–19, in the Skyline Ballroom North Pre-Function. Come and talk to us, especially at the poster reception on Tuesday from 5:15 to 7 pm.
TEIR: A Tiled Execution IR for Tensor Contractions
TEIR is an intermediate representation (IR) that describes a tensor contraction as loops around operations on small tiles. Our MLIR-based CPU compiler derives tiling, loop order and parallelization from a few hardware parameters without auto-tuning and maps the tile operations to LIBXSMM kernels. For 24 row-major TCCG contractions, the generated code is on average 1.5–1.9× faster than the next-best backend on CPUs from AMD, NVIDIA and Apple.
Tensor Contractions on AWS Trainium2
Each NeuronCore of Trainium2 multiplies matrices on a 128×128 systolic array. Fast kernels must fill the array’s rows and columns and hide data movement behind computation. Our Trainium2 compiler lowers a contraction to TEIR, picks the tiling with a performance model based on measurements and generates a kernel for the Neuron Kernel Interface. On two NeuronCores, the generated kernels reach up to 99% of the FP16 peak for matrix multiplications and up to 93% for four TCCG contractions.
VLIW Scheduling: Timing Is All You Need
The XDNA2 neural processing unit (NPU) reaches peak performance only if its VLIW cores issue a BFP16 matrix multiply-accumulate every cycle. Double-buffering the required 2×2 block of output tiles needs eight accumulator registers, but a core has only five. Our hand-scheduled kernel time-multiplexes the four tiles onto two registers and achieves 96% of peak on one core with L1-resident data. For a large matrix multiplication, the whole NPU reaches 21.4 BFP16 TFLOPS and saturates its DRAM bandwidth.


