Amazon has introduced Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI, providing a structured pipeline for training models that handle multi-step agentic tasks. While standard fine-tuning is typically suited for single-prompt responses, MTRL is designed to optimize sequences of actions where the outcome depends on cumulative decisions.
Training for Multi-Step Agentic Tasks
In many real-world applications, such as internal business workflows, there is no pre-existing “correct” sequence for an AI to imitate. Instead, the model must be trained to navigate a series of steps to achieve a specific result. MTRL addresses this by allowing developers to define a reward function that evaluates the outcome of an entire sequence. The system uses Amazon Bedrock AgentCore to host the environment where the agent performs tasks, calls the policy model, and eventually returns a reward upon completion.
To demonstrate the capability, Amazon used the game Wordle as a testing ground. Because the game requires multi-turn strategy based on evolving feedback, it serves as an effective proxy for more complex business processes. In this proof-of-concept, the training pipeline improved the solve rate of an open-weights 20B model from 61 percent to 84 percent.
System Infrastructure and Deployment
The MTRL pipeline on SageMaker AI is serverless, meaning users do not need to manually configure clusters or manage infrastructure sizing. The system provisions the necessary resources for training and sampling. Key components include:
- Policy Model: The base model currently supports options such as Amazon Nova 2 Lite, GPT-OSS-20B, Gemma 4 31B, and Qwen 3.6 27B.
- Training Mechanism: It utilizes Group Relative Policy Optimization (GRPO) to update Low-Rank Adaptation (LoRA) adapters based on reward variance across multiple rollouts.
- Monitoring: Integration with MLflow allows for real-time tracking of rewards and token usage throughout the training process.
Strategic Considerations
For organizations, MTRL offers a method to specialize small, cost-effective models for unique workflows that are not captured in general-purpose model training. Amazon emphasizes that datasets, agents, and trained models remain within the user’s AWS account and are not utilized to train base models. While the process provides flexibility, the company notes that reward design is critical, recommending an outcome-dominant reward structure with minimal shaping terms to ensure consistent performance.
As of the announcement, the capability is available in specific AWS regions, including US East (N. Virginia) and US West (Oregon), with pricing based on token consumption across prefill, sample, and training dimensions.
Source: aws.amazon.com
Original article: https://aws.amazon.com/blogs/machine-learning/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide/
Leave a Reply