Open R1: Update #3

Here’s an example of a straightforward query failing to return the long CoT:

User: What’s 1+1?

Assistant: 2. It’s one in all the best mathematical addition problems. 1 + 1 equals 2. This problem is commonly used as a riddle or puzzle, particularly in job interviews or online tests to find out a candidate’s ability to resolve problems quickly. While it could seem obvious to most individuals, it will possibly be a useful query for employers to evaluate critical considering and problem-solving skills. Moreover, it’s a vital concept in mathematics and forms the premise for more complex arithmetic operations. So next time someone asks you what 1+1 is, you may confidently state that it’s 2!

def add_numbers(a, b):
    return a + b


result = add_numbers(1, 1)
print("The sum of 1 + 1 is:", result)

On this solution, we define a function add_numbers that takes two parameters a and b and returns their sum. We then call this function with 1 and 1, storing the lead to the variable result, which is then printed. This demonstrates an easy option to compute and display the sum of two integers. The instance serves as a straightforward yet effective illustration of basic arithmetic operations in Python.

Nevertheless, if we prefill the assistant’s response with the specified behaviour emerges:

User: What’s 1+1?

Assistant: Okay, I want to reply the query “What’s 1+1?” Let’s examine. Oh right, 1 plus 1 equals 2. But wait, perhaps there’s more to it. Like, what if someone asks differently? Or perhaps considering other number bases? Like in base 10 it’s 2, but in binary it’s 10. Hmm, however the query is simple. The user probably expects the easy answer. So the right answer here is 2. Should I just state that? Yes, 1 + 1 is the same as 2. Is there anything you want to find out about this calculation?

This implies that when distilling reasoning traces into an existing post-trained model, one should implement prefilling by default in the ultimate model’s chat template. Incidentally, this can also be what DeepSeek does with their distilled models and the behaviour above often is the reason why. (It’s also amusing to invest whether hitting the “think” button on popular chat UIs is solely prefilling the assistant response 🙂).

Combining all these lessons produced OlympicCoder-7B, which matches the performance of DeepSeek’s distilled model and significantly improves on the bottom Qwen2.5-Coder one:

Lesson 5: Use 8-bit optimisers to scale large models with long context

Throughout the training of OlympicCoder-7B, we found DeepSpeed ZeRO-3 was sufficient to coach each model with 32k context on a single node of 8 x H100s. Nevertheless, once we tried scaling up our recipe to 32B, we bumped into a bunch of memory issues. Particularly, our runs would OOM once the context was scaled beyond 20k tokens, even on 16 nodes 😢. This wasn’t ideal as 20% of the CodeForces-CoTs traces are larger than 20k tokens, which implies they’d get truncated during training.

The foundation of the issue is that transformers and trl don’t yet support context parallelism, although see this issue to trace progress.

Within the meantime, we explored a wide range of memory saving techniques and located that combining FSDP with the paged_adamw_8bit optimizer allowed us to scale the context to 22,528 tokens: still not ideal, but only 9% of the info was truncated.

Updates

GRPO Updates

Recent progress has been made to enhance the implementation of GRPO in TRL, bringing enhancements that further boost efficiency, scalability, and resource utilization. Here’s a summary of an important changes for the reason that last update:

Generation Reuse

One among the fundamental bottlenecks of GRPO is similar as for any online method: generation takes time. A key option to make GRPO more sample-efficient is to reuse generated samples multiple times during optimization as an alternative of discarding them after a single use. This method was actually introduced a protracted time ago with PPO.

For GRPO, the variety of times a sample is reused is denoted as μ.

It’s now possible to reuse generated samples multiple times, significantly speeding up the method.

from trl import GRPOConfig

training_args = GRPOConfig(..., num_iterations=...)

Nevertheless, be cautious—if μ is simply too large, it will possibly negatively impact learning. Based on our experience, a superb balance is between 2 and 4.

Reward Weighting

When training a model, not all rewards are equally necessary. For instance, we might want the model to prioritize correctness over formatting, slightly than treating each elements equally.

To handle this, it’s now possible to assign different weights to different rewards, allowing finer control over the optimization process. By adjusting these weights, we are able to guide the model to focus more on the elements that matter most for a given task.

from trl import GRPOConfig, GRPOTrainer

def very_important_reward(completions, **kwargs):
    ...

def less_important_reward(completions, **kwargs):
    ...

training_args = GRPOConfig(
    ...,
    reward_weights=[0.9, 0.1],
)
trainer = GRPOTrainer(
    ...,
    reward_funcs=[very_important_reward, less_important_reward],
    args=training_args,
)

Other Enhancements

Several smaller but impactful improvements have been made to GRPO:

• PEFT + vLLM Integration – It’s now possible to make use of PEFT (Parameter-Efficient Positive-Tuning) and vLLM together, combining efficient fine-tuning with optimized inference for higher scalability.
• Gradient Checkpointing – This feature has been added to cut back memory consumption during training by recomputing certain activations as an alternative of storing them, making it possible to coach larger models.
• Optimized Selective Log Softmax Computation – A brand new method for computing log softmax has been introduced, reducing memory peaks during training.

Next Steps and Work in Progress

The present focus is on two key areas:

1. Improving Generation Speed – Further optimizations (like static caching) are being explored to make the generation process even faster.
2. Scaling GRPO to Multi-Node Settings – Work is underway to enable GRPO to scale across multiple nodes, making it possible to coach much larger models.

Open R1 Math-Dataset update

We have now further enriched our previously released OpenR1-Math-Raw dataset with recent metadata to enable more informed decision-making during filtering and verification. Specifically, we’ve added the next columns:

• reparsed_answers: We observed that many entries within the answer column were either improperly formatted in LaTeX or contained only partial answers. Moreover, since some questions are multiple-choice, each the right answer itself and the corresponding letter ought to be considered valid responses. To handle this, we re-extracted all answers from the solution column using the Llama-3.3-70B-Instruct model to make sure reparsed_answers includes each the right answer and, in multiple-choice cases, the corresponding letter as well. We consider this addition can be highly worthwhile to the community, improving each column grading accuracy and verification processes during GRPO.
• correctness: Running answer verification could be resource-intensive when counting on model based one. Due to this fact, we evaluated all solutions using Llama-3.3-70B-Instruct as a judge alongside with math_verify run against each the answer and reparsed_answers columns.

Experiment Details

To assist the community understand the impact of verification-based filtering on math datasets for SFT distillation, we conducted several ablation experiments. Starting with a randomly chosen pool of 200k samples, we created six distinct SFT datasets based on the next filtering rules:

1. no_restrictions (200k) – No filtering applied.
2. llama_verification (124k) – Samples that were graded as correct by LLAMA-70B verification.
3. math_verify_answer (88.7k) – Samples verified as correct by math_verify on the answer column.
4. math_verify_reparsed (101k) – Samples verified as correct by math_verify on the reparsed_answers column.
5. llama_verification_or_math_verify_reparsed (LorMV) (154k) – The union of datasets 2 and 4.
6. llama_verification_and_math_verify_reparsed (LandMV) (71.2k) – The intersection of datasets 2 and 4.

Training and Evaluation

For filtering in data-constrained settings, each precision and recall are necessary considerations. Due to this fact, we didn’t run each experiment with the identical token budget but slightly trained on single epoch over all data. For the model, we selected Qwen7B-Instruct and fine-tuned it with RoPE extension to a 32k context length and cosine schedule. To trace performance progression, we evaluated the models at every fortieth step on AIME-24, AIME-25, and MATH-500 using lighteval. The outcomes are summarized below:

Key Observations

• Verification significantly impacts early-stage performance. Filtering proved especially necessary throughout the first 40 steps. On the MATH-500 dataset, stricter verification methods delivered a considerable performance boost in early stages (e.g., no_restrictions scored 0.61 vs. LandMV at 0.72). Nevertheless, as training progressed, this performance gap diminished, and having more samples proved helpful—even when some contained errors.
• Training loss differed notably between datasets. Datasets filtered using math_verify exhibited consistently lower training loss. We assume that math_verify effectively identifies specific subsets of tasks (primarily numeric ones), whereas Llama-based verification or unfiltered datasets maintain a broader variety of information.
• Surprisingly, unfiltered datasets didn’t suffer severe degradation. Despite containing incorrect samples, the no_restrictions dataset maintained competitive performance over prolonged training runs.

Recommendations

Based on our findings, the optimal filtering method depends heavily on the training token budget. For shorter runs, stricter verification methods offer significant benefits. As a general advice, we recommend combining llama verification with math_verify—as done within the LorMV dataset.

The Reasoning Course

The Hugging Face learn team are working on accessible material on Reinforcement Learning, GRPO, and training reasoning models with TRL. It includes tutorials and demos from Maxime Labonne, Unsloth, and Marimo. Take a look at the reasoning course in the event you’re in search of a superb place to start on this fast paced field!

Community highlights

The previous few weeks have seen continued exploration of GRPO across a wide selection of tasks, together with several recent reasoning datasets that focus on domains which are broader than math and code. Here’s a few of the releases we found especially exciting:

GRPO explorations

• The optimisation wizards at Unsloth have further reduced the quantity of VRAM needed to coach GRPO models with LoRA, now right down to 5GB for a 1.5B model 🧙
• Kalomaze has written an excellent blog post on selecting good hyperparameters for GRPO. Additionally they observe that models smaller than 7B are likely to converge much slower, which means it’s best to explore recent ideas at these scales before concluding they don’t work.
• Hrishbh Dalal has shown which you can use GRPO to show LLMs to resolve Sudoku puzzles!
• Yutai Li and co-authors have published a excellent paper which shows that for smol models, it’s best to distill a mix of long and short CoT data from more powerful teachers.

Reasoning datasets

• KodKode have released a really large dataset of ~500k samples for programming. This looks like a wonderful complement to CodeForces-CoTs and we’re excited to coach some recent models on it!
• The team at GeneralReasoning have began releasing high-quality and diverse datasets like GeneralReasoning/GeneralThought-323K that cover each more domains and models than what’s publicly available. Additionally they have a pleasant website that allows you to explore the info, together with

What’s next?

With this update, we now have the fundamental pieces in place to finish Steps 1 and a pair of of our reproduction plan:

In the approaching weeks, we plan to focus heavily on:

• Perfecting the mix of distilled datasets to coach general-purpose reasoners.
• Scaling up GRPO to larger models like Qwen/Qwen2.5-Coder-32B-Instruct so we are able to derive R1-Zero variants.
• Combining reward signals from multiple domains like math and code, in addition to folding in reward models to attain non-reasoning data.