Roll out & verify
Sample multiple agent trajectories for one repository issue. Execute tests, then group attempts by their pass or fail outcome.
BSR / Software engineering agents
Beyond Binary Verifiable Rewards:Behavioral Self-Rewardingfor SWE Agents
Learn from how an agent debugs, even when the tests fail.
A richer training signal, anchored in executable verification.
Research preprint · 2026
1 Pennsylvania State University2 Mistral AI3 AG2AI4 Microsoft Research
14B + 32B model scales+2.6 pp 14B average accuracy · 400 steps+3.4 pp 32B Pass@3 · 270 steps
The intuition
Not all failures are equal.
Three attempts. The same test outcome. Different debugging quality.
ONE REPOSITORY ISSUETHREE AGENT TRAJECTORIESBehavioral signal enabled
AMeaningful progress
Inspect→Edit→TestFail
BPartial progress
Inspect→Edit→…Fail
CUnproductive exploration
Search→Repeat→RepeatFail
Compare & rank
Same task.
Same verifier outcome.
TRAINING REWARD
A
+0.15
B
0.00
C
−0.15
Useful contrast.
More to learn from.
Illustrative rewards for a three-failure group, using the main setting’s ±0.15 bound. Switch modes to see the learning signal that BSR adds.
Abstract & motivationA better way to learn
A better way to learn
from failed attempts.
Reinforcement learning with verifiable rewards provides reliable supervision for software engineering agents through executable tests. But binary rewards compress an entire debugging process into a single pass or fail—and cannot distinguish attempts that make meaningful progress from those that get stuck.
Behavioral Self-Rewarding (BSR) refines this signal using the agent’s own judgments. It compares trajectories from the same task with the same verifier outcome, ranks their observable debugging behavior, and maps those comparisons to small, bounded auxiliary rewards.
The result is additional learning signal from otherwise indistinguishable rollout groups, while every passing attempt still receives a higher reward than every failing attempt.
Read the full abstract
Reinforcement learning with verifiable rewards (RLVR) provides reliable supervision for software engineering agents through executable tests, but binary rewards fail to distinguish trajectories with substantially different debugging quality. We introduce Behavioral Self-Rewarding (BSR), a reward-refinement method that preserves executable verification as the source of correctness while using the agent’s own behavioral judgments to rank trajectories with the same verifier outcome. BSR converts pairwise trajectory comparisons into bounded auxiliary rewards, recovering learning signal from otherwise indistinguishable rollout groups. Experiments on repository-level software engineering tasks show that BSR consistently improves over verifier-only RL across Qwen3-14B and Qwen3-32B, with the largest benefit coming from distinguishing among failed trajectories. These results demonstrate that behavioral self-rewarding can effectively complement sparse verifiable rewards for training software engineering agents.
How BSR works
Verification for correctness.
Self-judgment for learning.
A reward-refinement layer for group-relative reinforcement learning.
Compare behavior
The policy judges pairs within an outcome group using the task, reference patch, and a rubric focused on observable debugging behavior.
Rank & refine
Fit task-local Bradley–Terry scores. A soft deadzone suppresses small differences; centering and bounding control the auxiliary reward.
Update the agent
Add the behavioral signal to the verifier reward and train. The main BSR setting refines only failures, keeping success at reward 1.
THE CORRECTNESS ANCHOR
R = Rverifier + rbehavior
Behavior adds nuance; executable verification defines success. Choosing amplitudes with a₀ + a₁ < 1 preserves the pass/fail reward ordering.
Experimental results
Stronger performance.
Across both model scales.
Trained on filtered R2E-Gym tasks. Evaluated on SWE-bench Verified.
+2.6pp
Average accuracy
32.1% 34.7%
+3.4pp
Pass@3
48.8% 52.2%
Learn from
the failures.
Refining only failed attempts performs best in the outcome-group ablation.
| Model | Steps | Method | Avg. accuracy ± std. | Max accuracy | Pass@3 |
|---|---|---|---|---|---|
| Qwen3-14B | 270 | Verifier-only RL | 31.0 ± 0.5 | 31.4 | 45.0 |
| BSR ours | 33.2 ± 1.4 | 34.8 | 46.6 | ||
| Qwen3-14B | 400 | Verifier-only RL | 32.1 ± 1.1 | 33.4 | 46.6 |
| BSR ours | 34.7 ± 1.0 | 35.6 | 48.6 | ||
| Qwen3-32B | 270 | Verifier-only RL | 34.4 ± 1.7 | 36.0 | 48.8 |
| BSR ours | 35.5 ± 0.8 | 36.2 | 52.2 |
Peak checkpoints within each predefined training budget; other metrics in each row use the same checkpoint. Evaluation uses temperature 1, a maximum of 50 steps, and three runs. “pp” denotes percentage points.
Learning curves
Evaluation on SWE-bench Verified throughout 270 training steps.
Understanding the gains
What makes the signal useful?
Failures carry the contrast.
Failure-only refinement reaches 33.2% average accuracy and 46.6% Pass@3, compared with 32.0% and 44.0% for pass-only refinement.
Judgment beats random noise.
Random auxiliary rewards with the same amplitude reach 30.9% average accuracy; BSR reaches 33.2%. Reward variance alone does not explain the gains.
Small differences need care.
The full deadzone and soft mapping reaches 33.2% average accuracy, compared with 32.0% when both are removed.
Ablations use Qwen3-14B. See the paper for full settings and training curves.
Citation
Build on this work.
If you find BSR useful in your research, please cite the paper.
Read the full paper@misc{wu2026bsr,
title = {Beyond Binary Verifiable Rewards:
Behavioral Self-Rewarding for {SWE} Agents},
author = {Wu, Yiran and Zhao, Andrew and
Chang, Yurui and Wu, Qingyun and Feng, Xuan},
year = {2026},
note = {Preprint},
url = {https://yiranwu0.github.io/projects/bsr/}
}