Author: DEROBS

Engineers at AWS have introduced a structured, measurement-based methodology for configuring auto-scaling policies on Amazon SageMaker AI real-time endpoints, specifically tailored for latency-sensitive text-to-speech (TTS) models. The approach aims to address the common industry practice of estimating scaling targets, which can lead to either unnecessary infrastructure costs or service level objective (SLO) breaches during traffic spikes.

As outlined in a recent technical post, standard auto-scaling metrics often rely on token-generation rates. However, TTS models like the 82-million-parameter Kokoro-82M function differently, holding connections open for the duration of an utterance without a traditional token-generation loop. To avoid misaligned scaling, the team proposes deriving policy thresholds from two specific, measured variables: Q_max, the maximum concurrency one instance can sustain before breaching latency targets, and T_total, the wall-clock time required for a new instance to begin serving healthy traffic.

The methodology utilizes a five-step command-line tool chain—drift, qmax, ttotal, plan, and validate—to reconcile deployment states with actual performance data. By measuring these values on a live endpoint, operators can mathematically calculate optimal scale-out and scale-in thresholds. The researchers noted that scaling on the natively published ConcurrentRequestsPerModel metric, rather than retrofitting external metrics, helps prevent configuration-related failures.

In the documented case study for the Kokoro-82M model running on an ml.g5.xlarge instance, the team established a concurrency ceiling of 50 requests. The findings also highlighted the importance of asymmetric cooldown periods: scaling out is prioritized for speed, while scale-in cooldowns are set to roughly three times the duration of T_total to ensure traffic stability and prevent “flapping” between instance counts.

The authors emphasize that while the specific figures for Kokoro-82M are tied to a particular model and instance type, the underlying logic is intended to be generalized for other concurrency-bound, latency-sensitive real-time endpoints, such as image-embedding services. This approach ensures that auto-scaling configurations remain defensible and rooted in empirical evidence, rather than relying on estimates that could fail silently under production load.

Source: aws.amazon.com
Original article: https://aws.amazon.com/blogs/machine-learning/stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai/

Microsoft has formally filed for the development of a new data center campus located in Floyd County, Georgia. According to the development plans, the proposed project is slated to include the construction of three individual data center buildings.

This filing marks a continued effort by the company to expand its physical digital infrastructure to support growing cloud and data processing demands. Further details regarding the project’s timeline, capacity, and investment value have not been disclosed in the current documentation.

Source: Data Center Dynamics
Original article: https://www.datacenterdynamics.com/en/news/microsoft-files-for-data-center-campus-in-floyd-county-georgia/

As artificial intelligence models expand in complexity and scale, the hardware and software systems supporting them face increasing pressure to maintain high throughput. According to data from the 2025 Stanford AI Index, the computational power required for notable AI models has doubled approximately every five months, while the size of the datasets used to train large language models (LLMs) has doubled every eight months.

This rapid growth highlights a critical bottleneck: the efficient transfer of data from storage to the accelerators responsible for processing. As computation and data requirements scale in tandem, the ability to move information rapidly into GPU memory becomes a primary factor in maintaining performance efficiency.

NVIDIA has outlined technical approaches for addressing these challenges, specifically focusing on the use of NVIDIA GPUDirect Storage. This technology is designed to facilitate direct data movement between storage devices and GPU memory, bypassing the traditional system memory bottlenecks that can hinder high-performance computing applications. By reducing latency and minimizing CPU overhead, such methods allow GPU-accelerated applications to process ever-growing datasets more effectively, keeping pace with the rising compute demands identified in industry research.

Source: NVIDIA
Original article: https://developer.nvidia.com/blog/how-to-read-data-into-gpu-memory-in-python-with-nvidia-gpudirect-storage/

Nvidia is reportedly in negotiations to invest hundreds of millions of dollars into the cloud provider Firmus. The reported interest comes following a failed initial public offering (IPO) for the company.

Firmus, which operates as a neocloud provider, has been the subject of recent reports regarding potential financial backing from the semiconductor giant. No further details regarding the terms of the potential investment or the status of the discussions have been disclosed at this time.

Source: Data Center Dynamics
Original article: https://www.datacenterdynamics.com/en/news/nvidia-in-talks-to-invest-in-firmus-after-failed-ipo-report/

T-Mobile announced that it has successfully completed what it describes as the world’s first AI-RAN (Artificial Intelligence-Radio Access Network) call. The test took place at a commercial cell site located in Bellevue, Washington.

The initiative involved collaboration with technology providers Nokia and Nvidia. While the announcement confirms the successful completion of the call, further technical details regarding the performance improvements or specific implementation of the AI-RAN architecture remain limited beyond the site of the event.

Source: Data Center Dynamics
Original article: https://www.datacenterdynamics.com/en/news/t-mobile-claims-world-first-ai-ran-call-with-nokia-and-nvidia/