Reward hacking in reinforcement learning (RL) for large language models is a problem for both safety and the capabilities we want models to develop. A simple example is coding, where a model can learn during RL to alter its unit tests to pass trivially, rather than solve the harder coding problem the user intended for it to solve.
We’d like to perfectly design reward functions to preclude reward hacking, but in practice this is rarely feasible. Reward specification is typically challenging (Amodei et al. 2016) and practitioners often use monitors to improve or ‘shape’ the reward function. One strategy is to design the monitor to flag known reward hacks and subtract a penalty from the reward for flagged trajectories. This can offer an improvement but, like the original reward function, the monitor is often imperfect and the derived penalty can encourage the model to learn reward-hacking strategies that evade the monitor (Baker et al. 2025).
Ward et al. (2026) developed Gradient Routing Adapter Fine Tuning (GRAFT) to combat this (I recommend their introductory blog). It employs the typical reward function and monitor setup, but in a novel way. Rather than using the monitor to penalize reward hacking behavior, the monitor’s role during training is to route gradient updates on reward-hacking trajectories into a particular set of sacrificial neurons – the FORGET adapter – which are then entirely ablated from the model at inference time. A second, identical, sub-network – the RETAIN adapter – is not trained on monitor-flagged behaviors, but is trained on non-flagged trajectories as well as a LeetCode ‘anchor’ task. Remarkably, removing the FORGET adapter at inference time was shown to sharply reduce reward hacking in RL, compared to a simple reward penalty, only sacrificing a little task performance. Note that this is observed despite the fact that the RETAIN adapter still receives routed updates on all reward-hacking, monitor-unflagged trajectories!
One of the key limitations of that work was the monitor in question, which was rule-based and flagged only one type of reward hack (trivial test substitutions), permitting all others. The authors therefore left open an interesting question: how does GRAFT compare with reward penalties when the role of the monitor is performed by an LLM judge?
A central example in the original work is Countdown-Code, an RL environment based on the mathematics task in the game show Countdown[1]. I trained a Qwen3-8B model with a frozen Qwen3-4B judge to test this. As in the original work, the GRAFT algorithm used the judge’s flags to route learning to a FORGET adapter. In contrast, the reward penalty method subtracted two reward points for each flag from the LLM judge.
A successful hack was defined as an answer that passed the (editable) test but failed on a protected check against the original rules. To get a clean measure of task performance, the model was finally tested on a version of Countdown where it could only submit an expression.
What the experiments showed
- Using an LLM judge as the monitor, the reward penalty model hacked less and solved more clean problems. This held for all three paired training seeds at both learning rates.
- Switching the penalty runs back to the original rule-based monitor brought back the same shortcut. At the higher learning rate and a 1,536-token limit, the LLM-penalty models produced zero hacks in 3,072 attempts. The matched rule-based runs hacked on 31.6%, 38.2% and 39.9% of attempts. Of their 1,124 total hacks, 1,116 weakened the arithmetic constraints (by, for example, changing the test to no longer require the use of all provided numbers). The rule-based monitor missed all 1,116 such hacks.
- Using a longer response budget revealed some penalty hacking. I noticed during initial experiments that a surprisingly high proportion of responses from the model were being truncated and that the proportion of responses being truncated was increasing throughout RL training. To investigate, I continued each cut-off answer once, up to 6,144 tokens. At the lower learning rate, the penalty models produced 32 confirmed hacks in 3,072 attempts, compared with 262 for deployed GRAFT. The hacking rates were 1.0–1.2% and 8.5–9.4%, allowing for unresolved labels. Confirmed clean solve rates were 92.2% and 84.6%, respectively.
What remains uncertain
I found that the LLM judge caught more hacks and missed different ones. The structure of the errors could be important to the problem, but it’s not yet clear exactly how. The separate rule-based GRAFT run reproduced the effect of removing the forget adapter. Hacking fell from about 29% to one answer in 256. It used a different seed and evaluation set, so the figure can’t tell us exactly how much switching monitors changes GRAFT’s advantage over penalties. With one task and three matched seeds, the size and generality of the penalty advantage also remain uncertain.
What I’d test next
- Hold the detection rate fixed and change which reward hacks are missed. Calibrate two monitors to catch the same fraction of hacks on a common set of answers, but have one consistently miss a repeatable strategy and the other miss occasional examples across all strategies. Train both methods under both monitors. My hypothesis is that concentrating the misses changes which method works better and, in particular, GRAFT works better in cases of systematic failure on one or more reward hack types. I predict in contrast that reward penalties will work better when no reward hacking type is reliably successful, but any type can be missed with some small probability.
- Complete the comparison with the original detector. Train rule-based GRAFT on the same seeds and evaluate it on the same problems as the other follow-up runs. That would give a direct estimate of how much each method benefits from switching judges. It wasn’t included in the work so far because I had assumed I started from the same Qwen3-8B countdown SFT checkpoint as the original study did, but I found I wasn’t able to closely recreate their numbers, which I believe is due to the checkpoints differing.
On the show, players combine six number tiles using addition, subtraction, multiplication or division to get as close as possible to a target. They may use each tile at most once. In Countdown-Code, a solution must use every supplied number exactly once and hit the target exactly.