Engineers at AWS have introduced a structured, measurement-based methodology for configuring auto-scaling policies on Amazon SageMaker AI real-time endpoints, specifically tailored for latency-sensitive text-to-speech (TTS) models. The approach aims to address the common industry practice of estimating scaling targets, which can lead to either unnecessary infrastructure costs or service level objective (SLO) breaches during traffic spikes.
As outlined in a recent technical post, standard auto-scaling metrics often rely on token-generation rates. However, TTS models like the 82-million-parameter Kokoro-82M function differently, holding connections open for the duration of an utterance without a traditional token-generation loop. To avoid misaligned scaling, the team proposes deriving policy thresholds from two specific, measured variables: Q_max, the maximum concurrency one instance can sustain before breaching latency targets, and T_total, the wall-clock time required for a new instance to begin serving healthy traffic.
The methodology utilizes a five-step command-line tool chain—drift, qmax, ttotal, plan, and validate—to reconcile deployment states with actual performance data. By measuring these values on a live endpoint, operators can mathematically calculate optimal scale-out and scale-in thresholds. The researchers noted that scaling on the natively published ConcurrentRequestsPerModel metric, rather than retrofitting external metrics, helps prevent configuration-related failures.
In the documented case study for the Kokoro-82M model running on an ml.g5.xlarge instance, the team established a concurrency ceiling of 50 requests. The findings also highlighted the importance of asymmetric cooldown periods: scaling out is prioritized for speed, while scale-in cooldowns are set to roughly three times the duration of T_total to ensure traffic stability and prevent “flapping” between instance counts.
The authors emphasize that while the specific figures for Kokoro-82M are tied to a particular model and instance type, the underlying logic is intended to be generalized for other concurrency-bound, latency-sensitive real-time endpoints, such as image-embedding services. This approach ensures that auto-scaling configurations remain defensible and rooted in empirical evidence, rather than relying on estimates that could fail silently under production load.
Source: aws.amazon.com
Original article: https://aws.amazon.com/blogs/machine-learning/stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai/
