RL post-training · research experiment RL 后训练 · 研究实验
CritPT-RL CritPT-RL
A complete RL post-training pipeline for scientific coding tasks, from data generation and reward design through GRPO training and evaluation.
一套面向 scientific coding tasks 的完整 RL post-training 流程，覆盖数据生成、reward 设计、GRPO 训练和评测。
任务
让模型交一份能运行的 Python 答案
任务形式很固定：模型读一道 scientific problem，返回一个可执行的 answer() 函数。看起来比开放式问答容易判分，但真正麻烦的是，代码能跑、格式正确和答案真的对，并不是一回事。
我想把这几个层次拆开，因此从数据、rollout、reward、checkpoint 到评测都自己接了一遍。
我做的
一条能反复改 reward 的实验链路
训练部分基于 verl 和 vLLM rollout，包含 checkpoint merge、评测和曲线整理。数据侧试过程序生成题、official-style prompt、从失败样本里挖 hard case，以及由 LLM 生成 teacher specification。
reward 也换过几轮：本地执行校验、语义代码判断、长度约束、严格 final-answer judge，再到 LLM judge。每一次改动都尽量保留对应配置和结果，而不是只留下最后一条命令。
结果
模型学会了“像答案”，但没有更会做题
后期 V / E 系列实验让输出更短、更规整，也更稳定地生成可执行的 answer()。但 official70 accuracy 没有提高。训练曲线里的进步，主要对应了 reward 容易看见的部分，而不是 benchmark 真正关心的语义正确性。
这个项目没有给出一个漂亮的 SOTA 数字，但它把一次 reward / eval mismatch 留下了足够清楚的证据。
仓库
哪些东西可以公开复现
仓库保留了数据生成、reward、训练配置、评测脚本、单元测试和筛选过的曲线。完整模型权重、原始 rollout 和远端算力环境没有公开；本地测试不需要模型权重或 API key。
THE TASK
Make the model submit runnable Python
The task has a narrow interface: read a scientific problem and return an executable answer() function. That makes scoring look simple, but runnable code, correct formatting, and a correct answer are three different things.
I wanted those layers to be visible, so I connected the path from data and rollouts through rewards, checkpoints, and evaluation.
THE BUILD
An experiment loop where rewards could be changed and compared
Training used verl and vLLM rollouts, with checkpoint merge, evaluation, and curve tooling. Data paths included programmatic tasks, official-style prompts, failure-mined hard cases, and LLM-generated teacher specifications.
I also iterated through local execution checks, semantic code judging, length shaping, strict final-answer judging, and LLM judges. Configs and results were kept together so an experiment left more than a launch command behind.
THE RESULT
The model learned to look more like an answer
Later V and E runs produced shorter, cleaner, more consistently executable answer() functions. Official70 accuracy, however, did not improve. The training curves rewarded what the verifier could easily see, not all of the semantic correctness the benchmark expected.
There is no neat SOTA number here. There is a well-documented reward/evaluation mismatch, which turned out to be more useful.
THE REPO
What can be reproduced publicly
The repository contains data builders, rewards, experiment configs, evaluation scripts, unit tests, and curated curves. Full model weights, raw rollouts, and private compute details are not published; local tests need neither weights nor API keys.