AI research company i14 has successfully implemented its Laplace 3B model, a Gated Linear Attention (GLA) architecture, on AWS Trainium2 hardware. By utilizing native PyTorch support via TorchNeuron Native, the team managed to bypass the conventional transformer limitations associated with key-value (KV) caches, which typically grow in size as context length increases.
Overcoming architectural constraints
Standard transformer models rely on softmax attention, which requires the KV cache to expand during inference, potentially increasing serving costs and limiting concurrent sequence handling. In contrast, i14’s Laplace 3B uses a recurrent state fixed at 29.5 MB per sequence, regardless of the input length. This design shift allows for O(T) training scaling rather than the O(T²) scaling seen in traditional transformers.
To adapt this architecture for AWS Trainium2, the AWS Generative AI Innovation Center (GenAIIC) team implemented several technical modifications. First, they replaced GPU-optimized Triton kernels with standard PyTorch operators tailored for the NeuronCore on-chip buffer. Second, they optimized the Muon optimizer by employing an all_to_all communication pattern, which reduced per-rank traffic by a factor of 24 and improved training throughput by 63%.
Performance and training outcomes
During training on a single trn2.48xlarge instance, the Laplace 3B model achieved a throughput of approximately 120,000 tokens per second. For inference tasks, the model served 11,544 output tokens per second across 1,280 concurrent sequences using the vLLM engine. The team noted that because the recurrent state remains constant, increasing prompt lengths from 2,048 to 8,192 tokens resulted in only a 0.5% increase in cost per output token.
Implications for AI deployment
This development highlights the potential for deploying custom model architectures on purpose-built hardware like Trainium2. By utilizing native PyTorch and aligning mathematical operations with the hardware’s internal memory architecture—specifically the NeuronCore’s State Buffer (SBUF)—i14 demonstrated that specialized designs can achieve predictable memory usage. This capability is particularly relevant for agentic AI applications that require long-range context retention without the burden of escalating memory requirements.
The i14 team noted that their engineering process also revealed the importance of profiling, as operator dispatch time, rather than HBM bandwidth, emerged as the primary bottleneck for inference. Future implementations of similar models may benefit from these refined techniques in compilation and communication overhead management.
Source: aws.amazon.com
Original article: https://aws.amazon.com/blogs/machine-learning/beyond-the-kv-cache-training-and-serving-i14s-laplace-3b-linear-attention-model-on-aws-trainium2-with-native-pytorch/
