Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents

Yiran Wu; Andrew Zhao; Yurui Chang; Qingyun Wu; Xuan Feng

Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents

Learn from how an agent debugs, even when the tests fail.
A richer training signal, anchored in executable verification.

Research preprint · 2026

Yiran Wu^1,*Andrew ZhaoYurui Chang¹Qingyun Wu²Xuan Feng³

¹ Pennsylvania State University² AG2AI³ Microsoft Research

^* Work partially done during an internship at Microsoft Research.

PaperRead the manuscript CodeGitHub repository LogsTraining runs

Models

BSR-14B BSR-32B

14B + 32B model scales+2.6 pp 14B average accuracy · 400 steps+2.34 pp 14B Live Lite Pass@3 · 400 steps

The intuition

Not all failures are equal.

Three attempts. The same test outcome. Different debugging quality.

One issue · Three trajectoriesWith BSR: rewards distinguish behavior

AMeaningful progress

InspectEditTestFail

BPartial progress

InspectEdit…Fail

CUnproductive exploration

SearchRepeatRepeatFail

Compare & rank

Same task.
Same verifier outcome.

A > B > C

Training reward

+0.15

0.00

−0.15

Different rewards provide a learning signal.

Illustrative rewards for a three-failure group, using the main setting’s ±0.15 bound. Switch modes to see the learning signal that BSR adds.

Abstract & motivation

Learning from failed attempts

Reinforcement learning with verifiable rewards (RLVR) provides reliable supervision for software engineering agents through executable tests, but binary rewards cannot distinguish attempts with different debugging quality. Behavioral Self-Rewarding (BSR) reuses the current policy to compare observable behavior among trajectories from the same task with the same verifier outcome. It aggregates pairwise preferences into task-local scores and maps them to bounded auxiliary rewards that preserve the verifier-defined pass/fail ordering. This adds supervision from already-collected trajectories without new environment interactions or a separately trained reward model. Experiments on SWE-bench Verified and SWE-bench-Live Lite show consistent gains across the evaluated Qwen3-14B and Qwen3-32B configurations, with additional improvements on Terminal-Bench 2.1.

How BSR works

From verified outcomes to behavioral rewards

A reward-refinement layer for group-relative reinforcement learning.

BSR framework: sample trajectories, verify outcomes, then reuse the policy to compare failed attempts using the task, traces, reference patch, and behavioral rubric. Pairwise preferences yield bounded rewards; the passing attempt retains reward 1. — BSR compares behavior within a verifier-outcome group and adds bounded rewards. The values shown are illustrative; the main setting uses a failure-reward bound of ±0.15.

Roll out & verify

Sample multiple agent trajectories for one repository issue. Execute tests, then group attempts by their pass or fail outcome.

Compare behavior

The same policy model judges pairs within an outcome group using the task, reference patch, and a rubric focused on observable debugging behavior.

Rank & refine

Fit task-local Bradley–Terry scores. A soft deadzone suppresses small differences; centering and bounding control the auxiliary reward.

Update the agent

Add the behavioral signal to the verifier reward and train. The main BSR setting refines only failures, keeping success at reward 1.

Preserving correctness

R = R_verifier + r_behavior

Behavior adds nuance; executable verification defines success. Choosing amplitudes with a₀ + a₁ < 1 preserves the pass/fail reward ordering.

Core method details Explore the comparisons, ranking, and reward mapping

Core method details

How BSR keeps self-judgment grounded

The judge adds contrast only where the executable verifier cannot.

Compare locally

Trajectories are compared only within the same task and verifier-outcome bucket. A good failure and a good success remain separate concepts.

Judge observable behavior

The post-rollout judge sees compact action traces, the task, verifier outcome, and reference patch. The acting policy never sees the reference patch.

Weight clear preferences

Pairwise judgments use slight, clear, or strong preference weights. Pair order is randomized to reduce positional artifacts.

Denoise and bound

Task-local Bradley–Terry scores are normalized, passed through a soft deadzone, centered, and bounded before they become rewards.

Pairwise judgmentspreferred trajectory + strength

Task-local rankingweighted Bradley–Terry scores

Noise controlnormalize + soft deadzone

Bounded rewardrᵢ = aᵧqᵢ, qᵢ ∈ [−1, 1]

Experimental results

Results across SWE benchmarks

Trained on 4,578 filtered R2E-Gym tasks. Evaluated on SWE-bench Verified and SWE-bench-Live Lite.

Qwen3-14B · 400 steps · Verified

+2.6pp

Average accuracy

32.1% 34.7%

Qwen3-14B · 400 steps · Live Lite

+2.34pp

Pass@3

11.33% 13.67%

Both model scales · Both SWE benchmarks

Consistent gains

BSR improves average accuracy and Pass@3 in every reported model and training-budget setting.

Main results Paper Table 1; all metrics in percent, higher is better.
Model	Steps	Method	SWE-bench Verified		SWE-bench-Live Lite
Model	Steps	Method	Avg. accuracy ± std.	Pass@3	Avg. accuracy ± std.	Pass@3
Qwen3-14B	270	Verifier-only RL	31.0 ± 0.5	45.0	7.11 ± 0.19	13.33
Qwen3-14B	270	BSR ours	33.2 ± 1.4	46.6	8.67 ± 0.88	15.33
Qwen3-14B	400	Verifier-only RL	32.1 ± 1.1	46.6	6.33 ± 0.34	11.33
Qwen3-14B	400	BSR ours	34.7 ± 1.0	48.0	8.11 ± 1.02	13.67
Qwen3-32B	270	Verifier-only RL	34.4 ± 1.7	48.8	7.67 ± 0.88	12.33
Qwen3-32B	270	BSR ours	35.5 ± 0.8	50.6	8.56 ± 0.51	13.67

SWE-bench Verified contains 500 tasks; SWE-bench-Live Lite contains 300. Evaluation uses temperature 1, a maximum of 50 agent steps, and three runs. Standard deviations are across runs; Pass@3 is the fraction of tasks solved by at least one of the three attempts. Values are from the reported checkpoints, rather than independently maximizing each metric. “pp” denotes percentage points.

Additional evaluation · Terminal-Bench 2.1

Paper Table 8: 89 tasks with three independent evaluation trials. BSR improves both reported metrics in all three settings.

Terminal-Bench 2.1 All metrics in percent; higher is better.
Model	Steps	Method	Avg. accuracy	Pass@3
Qwen3-14B	270	Verifier-only RL	7.12	14.61
Qwen3-14B	270	BSR	7.87	15.73
Qwen3-14B	400	Verifier-only RL	6.74	14.61
Qwen3-14B	400	BSR	8.24	15.73
Qwen3-32B	270	Verifier-only RL	4.87	8.99
Qwen3-32B	270	BSR	5.62	13.48

Learning curves

Training signal and SWE-bench Verified average accuracy throughout 270 training steps.

Mean critic score during training for Qwen3-14B. The BSR curve remains above the verifier-only baseline through most of training. — **Training signal**Mean critic score · EMA smoothed

Qwen3-14B average accuracy across 270 training steps. BSR peaks at 33.2 percent, compared with 31.0 percent for the verifier-only baseline. — **Evaluation**Average accuracy · SWE-bench Verified

Mean critic score during training for Qwen3-32B. BSR and the verifier-only baseline both rise over training, with BSR generally higher. — **Training signal**Mean critic score · EMA smoothed

Qwen3-32B average accuracy across 270 training steps. BSR peaks at 35.5 percent, compared with 34.4 percent for the verifier-only baseline. — **Evaluation**Average accuracy · SWE-bench Verified

Behavior analysis

How does BSR change the agent’s behavior?

Across 1,500 Qwen3-14B evaluation trajectories per method on SWE-bench Verified, the aggregate pattern is consistent with stronger investigation, follow-through, and verification.

InvestigateRead evidenceEditTest

Reference file viewed before first edit

~~73.3%~~74.8%

Search-to-edit conversion

~~46.5%~~49.4%

Post-edit test rate

~~30.1%~~31.1%

Average searches

~~4.74~~4.06

Average errors

~~13.38~~12.76

Average runtime

~~1,132 s~~1,069 s

Ablations

Three checks on where the gains come from

Qwen3-14B on SWE-bench Verified. Select any study. The view advances every 3 seconds and pauses while you hover or interact with it.

Behavioral contrast matters most among failures.

Failure-only BSR performs best and avoids the extra judgment cost of comparing successful trajectories.

33.2%average accuracy46.6%Pass@3

Training group	Avg.	Pass@3
Pass bucket only	32.0	44.0
Pass + fail buckets	33.1	44.6
Fail bucket only (BSR)	33.2	46.6

Structured judgment—not reward variance—drives the gain.

Random auxiliary rewards use the same amplitude as BSR but discard behavioral rankings. They do not improve the baseline.

+2.3 ppBSR vs. random, Avg.+3.8 ppBSR vs. random, Pass@3

Method	Avg.	Pass@3
Verifier-only baseline	31.0	45.0
Random reward	30.9	42.8
BSR	33.2	46.6

Denoising small preferences improves reward quality.

The deadzone alone raises average accuracy but lowers Pass@3. Combining it with softness improves both metrics over either simplified mapping.

+1.2 ppAvg. vs. no deadzone+1.0 ppPass@3 vs. no deadzone

Reward mapping	Avg.	Pass@3
No deadzone, no softness	32.0	45.6
Deadzone, no softness	32.6	44.0
Deadzone + softness (BSR)	33.2	46.6

1 / 3 Advances every 3 seconds

Citation

@article{wu2026beyond,
  title = {Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents},
  author = {Wu, Yiran and Zhao, Andrew and Chang, Yurui and Wu, Qingyun and Feng, Xuan},
  journal = {arXiv preprint arXiv:2610.12598},
  year = {2026}
}