Starting with an existing model, there are now many ways to improve reasoning capabilities through post-training. Some methods train the model to imitate target text and others train it using evaluations of its own outputs. In practice many pipelines combine methods of both types, and hybrid approaches, too, so this is just an organizing distinction and not a true taxonomy. We’ll discuss these methods at an abstract level – the technical details aren’t important for the story here, but I’ll link papers so you can take a look if you’re interested.
Learning from target text
Method Summary Any judgment that’s required, like checking which generated responses are acceptable, happens prior to the training step. By the time training begins, the model sees only the selected text and learns to imitate it.
Provided or teacher-generated text
In this strategy, you simply continue training using (something like) the ordinary next-token loss on new data that promotes reasoning. The challenge is largely in constructing high-quality demonstrations. Human reasoning traces are costly and difficult to scale, while synthetic traces depend on a sufficiently capable ‘teacher’ model. Examples include
Continued pre-training on human-written mathematics, code, proofs, etc. DeepSeekMath Corpus (Shao et al. 2024) is an example of such a dataset.
Chain-of-thought SFT[1] trains on complete worked solutions rather than final answers alone. When it’s done to a vanilla base model in order to warm up the model in advance of doing RL, you may see this referred to as “cold-start” SFT. DeepSeek-R1-Zero (DeepSeek-AI et al. 2025) shows that reasoning can emerge from RL without supervised warm-up. The DeepSeek-R1 pipeline, in contrast, begins by warming up the model with cold-start examples to establish a readable, human reasoning format before the RL stage.
Distillation, where a smaller or less capable ‘student’ model can learn directly from reasoning traces from a larger or more capable ‘teacher’ model. (See, e.g., Hsieh et al. 2023)
Self-generated and selected text
In some verifiable domains, it’s possible to automatically check correctness at scale. For example, some mathematical proofs can be formally represented and checked using “proof assistant” systems like Lean. Another example is code which can be automatically checked against a battery of unit, integration, or other tests.
It follows that in these domains we can use the models themselves to automate the construction of examples of reasoning using a simple recipe: (1) generate many completions to a question, (2) apply an acceptance rule to each completion, (3) train on the accepted ones. Many of these techniques can be applied iteratively. Starting with model \(\pi_k\), we create new training data from this model and perform SFT giving \(\pi_{k+1}\), and so on. Each new model therefore learns from whatever the acceptance rule allows through. Repeated rounds of the process can reinforce correct reasoning, but they can also reinforce flawed reasoning (provided it led to a correct answer).
Methods in this family include
Rejection-sampling fine-tuning. This is the name normally given to the most basic implementation of the above idea, since it is an example of a rejection-sampling algorithm in statistics. (See, e.g. DeepSeek-AI et al. 2025)
Self-taught Reasoner (STaR) (Zelikman et al. 2022). Similar to rejection-sampling, except when an answer is rejected, the model gets another chance to get it right through a process they call rationalization; the incorrect answer and (presumably) faulty reasoning are discarded, and a fresh model call is made using the original question along with the correct answer as a hint. Using this new hint, the model is asked to answer again. If it gets it right this time, then the new reasoning is kept as a positive example along with those examples that didn’t require a hint. Otherwise, if it still gets the answer wrong, the reasoning is thrown out.
Learned reward-model filtering. Not a training procedure itself, but a common technique used within the methods above. A reward model is simply another model, sometimes an LLM of the same type/family as the main LLM being trained, that is trained to help determine whether answers given by the LLM are correct, safe, or desirable in some other way. We can then use the verdict of this model within rejection sampling or top-K selection, for example, to aid in the process of filtering generated data down to an acceptable self-imitation training set. Reward models are particularly useful in areas where correctness can’t be checked unambiguously with unit tests, proof checkers, and so on. Since they’re learned models, they are inevitably fallible - more on this later. An example of a model trained (in part) using rejection sampling fine-tuning with a reward model is Llama 2 (Touvron et al. 2023).
Note that these approaches can suffer from a simple but profound weakness, depending on the nature of the problem and acceptance rule. In cases where the acceptance rule checks only the final answer, it approves not only correct reasoning but those outputs where incorrect reasoning has given rise to the correct answer. STaR would be an example of a technique that is particularly vulnerable to this. A further weakness is that many apparently verifiable domains are verifiable only with respect to the limited properties checked by the available procedure; if code has a flaw that lies outside of the detection of the battery of tests it’s put through, this won’t be detected and the example will be trained on.
Learning from evaluations
Method Summary The required judgment is part of the training signal. Rather than dictating which text is admitted for training, as in the previous section, here the evaluation itself is the key signal in updating the model.
Pairwise preferences
Consider training data of the form \((x, y^-, y^+)\), where \(x\) is the prompt and the \(y\)s are two generations from a model (perhaps the same model) given \(x\), and where we have the additional preference information that \(y^+\) is preferred to \(y^-\), often written \(y^+ \succ y^-\). These data might come from a human, another model, or an automatic check (in verifiable domains). An important detail is that the ordering is just a relative preference; \(y^+\) is preferred to \(y^-\), but it may still be wrong or undesirable.
A widely used method for learning from data of this kind is direct preference optimization (DPO) (Rafailov et al. 2024). DPO starts with two copies of the existing model. One copy will become the trainable model and the other is a frozen reference. Briefly, the trainable model is updated to increase the likelihood of the preferred response relative to the less-preferred response, measured against this reference. The reference acts as a regularizer, discouraging the model from changing too much. Unlike SFT, DPO doesn’t learn from the preferred response alone, but from the contrast between the preferred and rejected responses. In standard offline DPO, the preference pairs are collected in advance and held fixed during the training run, though there are iterative variants where the pairs can be refreshed using new model outputs.
Scalar rewards
Rather than comparing two responses directly, we can go response by response, assigning each a scalar reward. Starting with the current model, we generate responses to a collection of prompts, score them, and then update the model so that higher-reward responses become more likely relative to lower-reward ones. This is then repeated using the updated model. Algorithms such as PPO and GRPO (Shao et al. 2024) differ in how they translate the observed rewards into a model update, but the underlying idea is the same.
As we saw earlier, in some domains we have access to automatic checking. Applying reinforcement learning to these kinds of signals gets a special name: reinforcement learning with verifiable rewards, or RLVR. The same drawbacks as before apply; where the verification or testing fails to catch undesirable reasoning, it can be reinforced using these methods. DeepSeek-R1-Zero is an example of reasoning capabilities being developed through RL with automatically checked rewards (DeepSeek-AI et al. 2025).
In cases where no automatic verifier is available, the reward can instead come from a learned reward model, as is done in reinforcement learning from human feedback (RLHF) (Ouyang et al. 2022). Pairwise preference data, described above, is used to train a reward model that assigns a scalar score to each response. This reward model then serves as a scalable proxy for human preferences, and is used for model optimization. The crucial distinction is how the comparisons between outputs affect the model update. In DPO, they’re used directly in the loss function, whereas in RLHF they’re first used to train a reward model which the model is then trained against.
A downside to using learned reward models is that they’re imperfect proxies, and to some degree this can undermine the effectiveness of approaches like RLHF. A similar problem arises at test-time, where it’s common to guide search with a learned verifier.
References
For clarity: SFT (supervised fine-tuning) is often a very similar process to pre-training, in the sense that it uses the same loss function, or a similar autoregressive cross-entropy objective. It’s given the name ‘supervised’ because it is constructed as prompt-answer pairs, or prompt-(reasoning + answer) pairs, in contrast to the unstructured nature of pre-training on general text corpora.