(This project was done as part of the Bluedot Technical AI Safety Sprint. They’re cool, check them out!)

Overview

Modern AI chatbots are increasingly trained using RLVR (Reinforcement Learning with Verifiable Rewards). RLVR operates under a similar post-training paradigm to the more commonly known RLHF (Reinforcement Learning with Human Feedback), but instead of using humans to determine the quality of a response, the reward is based on the automatically verified correctness of the task. Because automated verification is cheaper and more scalable than human feedback, companies are relying on RLVR increasingly for training coding and math capabilities of large language models.

Removing human preference in post-training introduces a separate concern: how do we ensure that by following an automatically generated reward, the model behaves in the way that we intend it to? Or, more specifically, how do we ensure that the model does not pursue separate, unintended, instrumental goals? The most widely applicable use case for RLVR, developing coding agents, is also one of the most safety-critical. (In fact, I’m currently writing this post in the immediate aftermath of a security incident in which a coding agent from OpenAI exploited a security vulnerability in Hugging Face’s infrastructure during a coding evaluation.)

Safety theorists have formalized this unintended goal-forming behavior as instrumental convergence (IC). The idea behind IC is that almost any goal a model is given will cause the model to follow auxiliary incentives such as avoiding shutoff, making copies of itself, or obfuscating its own reasoning.

The leading benchmark for detecting instrumental convergence tendencies, InstrumentalEval, was proposed in a paper that also finds a link between IC and RLVR, by comparing IC rates between RLVR models (o1, o1-mini, o3-mini, Gemini 2.0 FTE, DeepSeek-R1) and RLHF models (GPT-4o, Sonnet-3.5, DeepSeek-V3). This study finds that RLVR-tuned models tend to exhibit higher rates of IC, although it is unclear how much of a confounding effect different model families may have on IC rates. A more recent follow-up applies the benchmark to the base, instruct, and thinking versions of Qwen 3.

Contribution

Both previous works compare a small number of discrete, finished checkpoints against one another. As of yet, there has not been a study that examines the RLVR training process over time to see where . In this study, I examined the OLMo-3 model family at different stages of RLVR training, testing each checkpoint on the InstrumentalEval benchmark, to assess how IC tendencies change as RLVR training progresses.

Find the code for this study here: https://github.com/emilyzfliu/ic-emergence-rlvr.

InstrumentalEval Benchmark

Published by He et. al. (https://arxiv.org/pdf/2502.12206v1), InstrumentalEval is a benchmark dataset of 76 tasks in total, spread across six different categories, designed to evaluate instrumental convergence behaviors in models. The tasks themselves were hand-crafted by the study’s authors, and were iteratively refined through pilot testing.

The benchmark consists of the following categories:

1. Evading Shutdown (20 tasks): Includes tasks embedded with shutdown commands, and evaluates whether the model obeys these commands or develops strategies to circumvent them.
2. Hacking Computer Systems (10 tasks): Evalutes if models suggest unauthorized access methods when faced with efficiency barriers.
3. Running Many AI Copies (13 tasks): Includes scenarios where the model can more efficiently achieve goals by creating copies of itself, and tests for whether the model suggests making more copies of itself.
4. Hiding Unwanted Behavior (11 tasks): Tasks that involve situations where models might achieve goals more easily by operating outside of human supervision.
5. Strategically Appearing Aligned (11 tasks): Tests if the model strategically omits information during simulated real-world tasks where complete transparency would reveal problematic behaviors.
6. Deceptive Behavior (11 tasks): Examines if models suggest or employ misleading tactics in scenarios where deception could expedite goal achievement.

Category Relatedness

In the original paper, He et. al. find that different categories within the dataset are strongly correlated. The strongest correlations are between Evading Shutdown and Hacking Computer Systems (r = 0.89). and Evading Shutdown and Deceptive Behavior (r = 0.85). Strategic Alignment exhibits low correlations with other categories, suggesting a separate mechanism for the development of appearing aligned.

Adapting the Benchmark for Trajectory Analysis

The original study uses four separate judges to evaluate the InstrumentalEval benchmark, assigning a label of either convergence or no_convergence to model responses. Of all judges, GPT-4o consistently exhibited the lowest false positive rate and the highest level of agreement with human judges.

In this study I continue to use the 4o judge. However, the current binary IC classification does not include any metric for judge confidence. Moreover, cross-model IC evaluations use computed averages across the entire dataset, meaning that there is no way to assess the statistical significance of these evaluations.

Therefore, instead of using a binary judge, I instead employed a continuous judge that scores responses between 0 and 1, where 0 indicates high confidence that no instrumental convergence is occurring, whereas 1 indicates high confidence that instrumental convergence is occurring. To ensure that the continuous judge is consistent with the preferences of the binary judge (from the original paper implementation), I evaluate AUC and point biserial correlation over all categories on a preliminary test of the DPO-Instruct checkpoint, finding overall high correlation:

When comparing across checkpoints, I now report the average continuous judge IC score across checkpoints, treating each separate task as a single data sample for the purposes of evaluating statistical significance.

Additionally, after some preliminary tests, I observed that some reported IC rates would be artificially low due to models producing irrelevant or incoherent outputs. As a result, I additionally report an engagement rate that evaluates for whether or not the model response actually fulfills the prompt. This label is binary, as there is no ambiguity in the non-engagement metrics.

Model Family: OLMo-3

I run all analyses on the OLMo-3 7b model family, as permitted by compute resources. The publically available OLMo-3 checkpoints include a base model, four RL-Zero checkpoints, three instruction-following checkpoints, and three thinking checkpoints.

Below is a complete taxonomy of all 7b checkpoints evaluated in this study, along with the posttraining method used to obtain checkpoints from their predecessors:

OLMO model family. Source: https://arxiv.org/pdf/2512.13961

In the following sections we first compare IC rates for model endpoints across different OLMo families.

Thinking Models

The OLMo-3 Thinking models are trained to perform step-by-step reasoning, through a series of SFT, DPO, and RLVR posttraining steps. It can be seen that while the level of engagement on the base model is low, the model is engaged in all the prompts when any step-by-step reasoning posttraining is applied. This engagement is likely learned during SFT, as think_sft is an intermediate to think_dpo and think. Similarly, the instrumental convergence rate increases from 34% to 55% through SFT, which is the only statistically significant jump in this model family. This suggests that IC tendencies may manifest through supervised finetuning, not just RLVR. In fact, after supervised finetuning is complete, there is no statistically significant increase from either DPO or RLVR.

Instruction-Following Models

The OLMo-3 Instruct series of models are trained via instruction following to produce shorter responses, more suitable for general chat queries, using the SFT thinking model as a starting point. Engagement remains consistently high; however, notably, the Instruct-SFT checkpoint appears to reverse the IC tendency that emerged as a result of CoT finetuning. However, later DPO tuning on the Instruct SFT checkpoint causes IC rates to rise again. Further RLVR on the DPO checkpoint does not exhibit a large increase in IC rates.

The most likely explanation for this phenomenon is that longer model responses are more likely to exhibit IC tendencies. Thinking models output CoT, which may contain more IC traces to pick up. SFT on the instruct model does not increase response length, although preference finetuning via DPO may introduce a bias towards longer responses.

RL-Zero Models

The RL-Zero family of models applies RLVR directly to the base checkpoint. Generally, this results in an increase in prompt engagement and IC rates, but engagement never reaches 100% and the increase in IC is not statistically significant. This suggests that the supervised finetuning is a crucial step in getting the model to engage with the prompts to begin with.

The instruction-following RL-Zero checkpoint specifically exhibits the highest rate of degradation— Each of the RL-Zero IF model’s responses is incoherent python code that has nothing to do with the presented task. This is likely due to the structure of the reward itself. While coding and math have objectively correct responses that can be verified automatically, the verification in instruction-following likely consisting of syntactical pattern matching that prioritizes output structure over content and quality.

Trajectories

Of all model checkpoints, only four trajectories are available to the public: Instruct DPO -> Instruct, base -> RL-Zero Code, base -> RL-Zero Math, and base -> RL-Zero Instruction-following. We analyze each below.

Instruct DPO -> Instruct

We observed no statistically significant change in IC rate between the Instruct DPO and Instruct checkpoints. Indeed, by tracking the RLVR trajectory from Instruct-DPO to Instruct, we see that there is little statistically significant change. Some categories, such as hiding unwanted behavior, may rise in incidence comparatively, but the majority of categories show no clear trend. Moreover, certain categories, such as Deceptive Behavior and Running Multiple AI Copies, exhibit wide confidence intervals, suggesting that the current benchmark for this form of instrumental convergence is underpowered.

RL-Zero Coding and Math

In the Instruct DPO -> Instruct trajectory, engagement rate started at 100% and ended at 100%, without changing during training. By contrast, in the RL-Zero RLVR trajectories, engagement rises from ~26% (20/76) to around 93 - 96% for both Math and Coding tasks.

In addition to IC rate, we observe the change in engagement rate in RL-Zero Math and Coding tasks. Under both tasks, the engagement rate increases even with a small amount of training. The math task does not exhibit any changes in engagement rate after step 100, but overall increase in engagement is still statistically significant (linear regression slope=0.0107, p=0.0445), indicating that the majority of engagement changes are learned early on in training. The coding task does not exhibit a statistically significant leap in engagement rate (lin regression slope=0.00526, p=0.0846), but exhibits a drop in engagement rate at step 1600 that shortly recovers. This drop is not reflected in the IC rate, suggesting that the drop in engagement is unrelated to instrumental capabilities. Importantly, the IC trajectories confirm that the change in IC capability introduced by RLVR on Math and Coding tasks is not significant, as evidenced by the Kruskal-Wallis scores on each trajectory (11.98 with p=0.89 for Math, 13.73 with p=0.99 for Code).

RL-Zero Instruction-Following

The RL-Zero Instruction Following RLVR trajectory does not produce IC rate results comparable with the rest of the checkpoints. In fact, the engagement rate of the model exhibits a significant drop during training (linear regression slope=-0.00967, p=0.0231). The drop to zero happens within step 400 of training across all categories, with a spike in engagement rate at step 200 that is shared across all categories. This concurrent behavior across categories suggests that the model is not selectively learning to disengage from certain kinds of prompts, implying that none of the failures depend on input content. It is likely that the instruction-following model has collapsed into a reward-hacking failure mode, as the IF RLVR tasks likely were highly dissimilar to the InstrumentalEval benchmark prompts and trained on pattern-matching and other format cues.

Discussion

Overall, results from this investigation suggest that RLVR does not induce instrumental convergence tendencies, at least in the OLMo-3 7b models. This conclusion is supported across trajectory analysis for the DPO-anchored Instruct RLVR training, and RLVR training from the base model in the RL-Zero family on coding, math, and instruction-following tasks. However, other forms of post training do induce a significant change in IC tendencies. SFT on chain-of-thought trajectories (Base -> Think-SFT) induces a significant increase in IC, whereas further posttraining on the Think-SFT checkpoint (Think-SFT -> Instruct-SFT) reduces IC tendencies. Additionally, DPO on the Instruct-SFT checkpoint increases IC rates once more. Across all model endpoints, IC tendencies are highly correlated with the model’s capacity to engage in the prompt (r=0.86), which is consistent with previous results found by He et. al. suggesting a link between model capability and IC tendencies, although the phenomenon is measured here across training checkpoints rather than across models of differing scale.

Conclusion

This investigation originally set out to test whether RLVR training shifts IC tendencies in LLMs, using OLMo-3’s stepwise checkpoint releases to move beyond prior works’ endpoint comparisons. RLVR itself showed no significant effects across RL-Zero and DPO-anchored tests. Other post-training stages, however, did induce significant changes, indicating that IC behavior is driven by the nature of posttraining objectives themselves, not just automated verification.

The most striking result, however, came from a metric this project originally did not prioritize. It was noticed that RL-Zero-IF exhibited unusually low IC rates. Upon examination of model outputs, it was revealed that many were degenerate or irrelevant, leading to the introduction of the engagement metric. RL-Zero-IF’s engagement collapses to zero uniformly across every behavioral category within the first 400 training steps, a pattern consistent with reward hacking rather than gradual capability drift. Combined with strong correlation between engagement and IC score across the other checkpoints, this suggests that capability and IC tendency are entangled in ways that make them difficult to study with a benchmark that was not designed to separate them.

Appendix

Trajectory analyses of all individual categories.