Optimizing multilingual reasoning in open-weight models with Amazon SageMaker HyperPod

<p>A new technical analysis from Amazon Web Services (AWS) details a framework for fine-tuning open-weight reasoning models to overcome language-specific performance gaps. The report focuses on the challenges of ensuring models maintain correct chain-of-thought reasoning in non-English languages, using the GPT-OSS-20B model as a primary test case on Amazon SageMaker HyperPod.</p><h2>The multilingual reasoning challenge</h2><p>Researchers noted that base models often default to English even when prompted in other languages such as French, German, Spanish, or Italian. In initial evaluations, GPT-OSS-20B demonstrated a 22 percent success rate for reasoning in a requested language, with nearly all correct responses originating from English-language test cases. This mismatch presents a significant hurdle for teams deploying AI to non-English speaking users.</p><h2>SFT and GRPO: A balanced approach</h2><p>The AWS team utilized a combination of supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) to address these deficiencies. The findings suggest that SFT and GRPO serve complementary roles: SFT is essential for teaching a behavior the model does not yet possess, while GRPO acts as a refinement tool for behaviors the model can already perform intermittently.</p><p>In the GPT-OSS-20B experiment, SFT successfully increased the model’s ability to reason in the target language to 100 percent. However, answer-language accuracy reached 62 percent. The team then applied GRPO, which provided an additional eight-point improvement to answer-language accuracy without degrading the reasoning performance established during SFT.</p><h2>Infrastructure and patterns</h2><p>The training stack leveraged Amazon SageMaker HyperPod orchestrated via Amazon Elastic Kubernetes Service (Amazon EKS). The process incorporated the PyTorch TRL library and utilized Low-Rank Adaptation (LoRA) to maintain parameter efficiency. To validate their findings, researchers ran a control experiment on the Qwen2.5-7B model, testing both the multilingual task and the GSM8K benchmark. The results confirmed that GRPO only yields performance gains when there is sufficient 'headroom'—situations where the model exhibits a variety of performance levels—reinforcing that GRPO is ineffective on tasks where a model is already performing at peak accuracy.</p><p>Ultimately, the AWS findings suggest a practical decision framework for developers: use SFT to establish missing capabilities, and reserve GRPO for refining behaviors that the model already exhibits at a basic, albeit inconsistent, level. Before committing compute resources to GRPO, teams are encouraged to verify that the target task possesses the necessary performance variance to allow for meaningful reinforcement.</p>

Source: aws.amazon.com
Original article: https://aws.amazon.com/blogs/machine-learning/fine-tuning-gpt-oss-20b-for-multilingual-reasoning-on-amazon-sagemaker-hyperpod/