Updating Amazon SageMaker AI real-time endpoints—such as modifying instance types or environment variables—can often result in deployment failures when using AWS CloudFormation or the AWS Cloud Development Kit (CDK). This challenge arises because SageMaker requires the deregistration of scalable targets before an endpoint configuration can be updated. When these processes are managed natively, CloudFormation often loses the ability to track Application Auto Scaling resources, leading to stuck states or endpoints running without auto-scaling capabilities.
Solving Orchestration Challenges
To address these limitations, a reusable CDK construct has been developed that utilizes custom resources to orchestrate the auto-scaling lifecycle during endpoint updates. Instead of relying on native resources alone, this solution employs three custom resources to manage the transition automatically:
- Deregister: Removes the scalable target before the endpoint update begins to prevent conflict.
- Register Scalable Target: Monitors the endpoint status and registers the target once it reaches an InService state.
- Attach Scaling Policy: Waits for the target to be ready before applying the designated scaling policy.
The construct uses a ConfigHash property, which ensures that CloudFormation triggers the necessary orchestration sequence whenever a change to the endpoint configuration or scaling parameters is detected. By leveraging the asynchronous completion pattern of the CDK provider framework, these operations can accommodate the time required for endpoints to initialize, preventing issues associated with standard Lambda timeouts.
Implementation and Integration
The solution provides a robust mechanism for developers to define endpoint parameters—such as instance types and model data URLs—alongside auto-scaling settings like minimum and maximum capacity within a single configuration. This approach ensures that updates are treated as a single, orchestrated event, reducing the need for manual intervention via the AWS CLI or the management console.
For organizations looking to integrate this logic into existing projects, the construct can be implemented by adding the required source files directly into a project and wrapping the existing endpoint definition. This method provides a cleaner, more resilient deployment pipeline for machine learning workloads, ensuring that infrastructure remains consistent across environment updates.
Source: aws.amazon.com
Original article: https://aws.amazon.com/blogs/machine-learning/orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk/
Leave a Reply