GRPO Fine-Tuning: Train Reasoning Into Small LLMs

If you have spent any time in the LLM trenches lately, you have noticed the post-training conversation move on from supervised fine-tuning. The new center of gravity is reinforcement learning — and inside that, the technique that put DeepSeek-R1 on the map: Group Relative Policy Optimization (GRPO). It is the recipe behind every recent reasoning model that can hold its own on math, code, and multi-step logic at fractions of the size of frontier giants.

The good news: you can run GRPO on a single consumer GPU. The better news: the Hugging Face trl library now ships a production-ready GRPOTrainer, and pairing it with Unsloth gives you 17B-parameter capacity on a 15 GB card. This guide walks through fine-tuning Qwen2.5-0.5B-Instruct on grade-school math problems so the model learns to think, not just imitate. By the end you will have a working reward function, a trained checkpoint, and a clear mental model of why GRPO works.