Skip to content

Training Agents: Reward and Credit

Hands-on

Full runnable recipe: training-agents/ in the companion cookbook — real DPO preference-pair construction from real completions, plus exact GRPO and DAPO formulas run on toy reward groups so their real documented failure mode and fix are actually visible in the numbers. No GPU, no gradient update — per this site's own standing decision, training content stays conceptual plus a CPU-only toy.

TL;DR — quick revision
  • GRPO replaces a critic with the group itself. DeepSeekMath's own framing: "GRPO foregoes the critic model, instead estimating the baseline from group scores" — rewards for a group of sampled completions to the same prompt are "normalized by subtracting the group average and dividing by the group standard deviation," and that normalized value becomes every token's advantage.
  • A real, exact run of that formula reproduces GRPO's own documented failure mode. A group where every sample got the identical reward produced an advantage of exactly zero for all eight samples — DAPO's own paper names this the "gradient-decreasing problem": "if all outputs of a particular prompt are correct and receive the same reward, the resulting advantage for this group is zero. A zero advantage results in zero policy gradients."
  • DAPO's fix, run for real on the same degenerate group, is exactly what the paper describes: over-sample and filter out any group whose accuracy is exactly 0 or exactly 1, keeping only groups with genuine variance — "leaving all prompts in the batch with effective gradients."
  • Outcome reward and process reward answer different questions, with a measured real gap between them. "Process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset" — a process-supervised model reported solving 78% of a representative MATH subset. A real toy repro shows exactly why: outcome-only credit can't tell a uniformly-bad trajectory from one where only the middle step failed; step-level credit can.
  • DPO turns preference pairs directly into a classification loss, no separate reward model or RL loop needed — "solve the standard RLHF problem with only a simple classification loss," reported to match or exceed PPO-based RLHF while being "stable, performant, and computationally lightweight." A real repro constructed one such pair from two real, independently sampled completions and a real judge call.

GRPO: the group is the baseline

Proximal Policy Optimization (PPO), the RLHF-era default, needs a learned critic (value function) to estimate how much better or worse an action was than expected — a second model, trained alongside the policy, adding real memory and instability cost. DeepSeekMath's Group Relative Policy Optimization sidesteps that entirely: "GRPO foregoes the critic model, instead estimating the baseline from group scores." Sample a group of completions to the same prompt, score each one, and use the group's own statistics as the baseline — "rewards are normalized by subtracting the group average and dividing by the group standard deviation," and "sets the advantages of all tokens in the output as the normalized reward." No critic, no separate value network — the group tells you, relative to itself, which samples were actually better.

import statistics


def grpo_advantages(rewards: list) -> list:
    """Exactly DeepSeekMath's formula: subtract the group mean, divide by
    the group standard deviation. No critic model, no value network --
    the group itself is the baseline."""
    mean = statistics.mean(rewards)
    std = statistics.pstdev(rewards)
    if std == 0:
        # The real, documented failure: DAPO's own paper calls this the
        # "gradient-decreasing problem" -- a zero-variance group produces
        # a zero advantage for every sample, and zero advantage means
        # zero policy gradient from this group, regardless of group size.
        return [0.0 for _ in rewards]
    return [(r - mean) / std for r in rewards]


def grpo_demo() -> dict:
    mixed_group = [1, 1, 0, 1, 0, 0, 1, 0]  # a genuinely hard prompt -- real variance across 8 samples
    degenerate_group = [1, 1, 1, 1, 1, 1, 1, 1]  # a prompt the model already always gets right

    return {
        "mixed_group": {"rewards": mixed_group, "advantages": [round(a, 3) for a in grpo_advantages(mixed_group)]},
        "degenerate_group": {"rewards": degenerate_group, "advantages": grpo_advantages(degenerate_group)},
    }

A real run — the exact formula, a real mixed group, and a real degenerate one

Input — a group of 8 binary rewards for a genuinely hard prompt (a mix of correct and incorrect samples): [1, 1, 0, 1, 0, 0, 1, 0].

Output — real computed advantages: [1.0, 1.0, -1.0, 1.0, -1.0, -1.0, 1.0, -1.0]. Every reward-1 sample gets a positive advantage, every reward-0 sample gets a negative one — exactly the group-relative signal the formula is supposed to produce, computed for real, not asserted.

Input — a second group, for a prompt the model already always answers correctly: [1, 1, 1, 1, 1, 1, 1, 1].

Output: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]. Every advantage is exactly zero — not a bug in this recipe's code, a real, exact reproduction of the mechanism DAPO's paper names and explains next.

DAPO: the gradient-decreasing problem, and the fix

That all-zero result above isn't a corner case nobody worries about — it's common enough in practice that DAPO built a named fix for it. The paper's own diagnosis: "Existing RL algorithm suffers from the gradient-decreasing problem when some prompts have accuracy equal to 1. For example for GRPO, if all outputs of a particular prompt are correct and receive the same reward, the resulting advantage for this group is zero. A zero advantage results in zero policy gradients, shrinking the magnitude and increasing the noise sensitivity of the batch gradient, thereby degrading sample efficiency." A model that's already mastered a prompt (or one it always fails, symmetrically) contributes nothing to training from that prompt at all, while still costing a full group of real sampling compute to discover that.

DAPO's fix, dynamic sampling, is exactly what it sounds like: "we propose to over-sample and filter out prompts with the accuracy equal to 1 and 0 ... leaving all prompts in the batch with effective gradients and keeping a consistent number of prompts." Keep sampling until you find groups with real internal variance, and only train on those. This is one of DAPO's four named techniques (alongside Clip-Higher, Token-Level Policy Gradient Loss, and Overlong Reward Shaping), and the paper reports the combination reaching 50 points on AIME 2024 with a Qwen2.5-32B base model — ahead of DeepSeek-R1-Zero-Qwen-32B's 47 points, using only half the training steps.

def dynamic_sample(candidate_groups: list) -> dict:
    """DAPO's actual fix: over-sample prompts, then filter out any group
    whose accuracy is exactly 0 or exactly 1 -- exactly what the paper
    describes -- keeping only groups that will produce a real gradient."""
    for group in candidate_groups:
        accuracy = sum(group) / len(group)
        if 0 < accuracy < 1:
            return {"kept_group": group, "accuracy": accuracy, "advantages": [round(a, 3) for a in grpo_advantages(group)]}
    return {"kept_group": None, "accuracy": None, "advantages": None}


def dapo_demo() -> dict:
    # Simulates re-sampling the same prompt: the first two attempts happen
    # to be degenerate (all-correct), the third has real variance -- DAPO
    # keeps sampling until it finds a group that isn't.
    candidate_groups = [
        [1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1],
        [1, 0, 1, 1, 0, 1, 0, 1],
    ]
    result = dynamic_sample(candidate_groups)
    return {"candidate_groups": candidate_groups, "resampling_result": result}

A real run — the exact degenerate group, filtered and replaced

Input — three candidate groups, simulating repeated sampling attempts on the same prompt: two degenerate ([1,1,1,1,1,1,1,1], twice) and one genuinely mixed ([1,0,1,1,0,1,0,1]).

Output: the filter correctly skipped both degenerate groups (accuracy exactly 1.0, per DAPO's own stated condition) and kept the third — accuracy 0.625 — computing real, non-degenerate advantages from it: [0.775, -1.291, 0.775, 0.775, -1.291, 0.775, -1.291, 0.775]. The same formula that produced all-zeros above now produces a real, usable gradient signal, because the input it was given this time actually had variance to normalize against.

Outcome reward vs. process reward

Both are answers to "what should the model be rewarded for," and they disagree about the unit that gets scored. Outcome supervision scores the final result — did the proof conclude correctly, did the task actually succeed. Process supervision scores each intermediate step on its own merits. PRM800K's own comparison is direct: "process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset," with a process-supervised model reported solving 78% of a representative MATH test subset — and the cost of that improvement is real too: 800,000 human step-level labels went into building it. Process reward models aren't free to build well, either — a later paper's own finding: Monte-Carlo-estimated PRM labels are flawed, and consensus filtering across multiple estimates is needed to get usable step-level signal at all.

def outcome_only_credit(step_rewards: list, final_outcome_reward: float) -> list:
    """The naive approach: the single final-outcome reward gets broadcast
    as the credit for every step in the trajectory, regardless of which
    step actually caused the outcome."""
    return [final_outcome_reward for _ in step_rewards]


def credit_assignment_demo() -> dict:
    # A toy 3-step trajectory: steps 1 and 3 were genuinely fine, step 2
    # was the actual mistake that caused the task to fail.
    step_labels = ["step 1 (correct sub-action)", "step 2 (the actual mistake)", "step 3 (correct sub-action)"]
    true_step_quality = [1, -1, 1]  # ground truth, for comparison only -- not seen by either method
    final_outcome_reward = -1  # the trajectory failed overall, because of step 2

    outcome_credit = outcome_only_credit(true_step_quality, final_outcome_reward)
    process_credit = true_step_quality  # a process/step-level reward model scores each step on its own merits

    return {
        "step_labels": step_labels,
        "outcome_only_credit": outcome_credit,
        "process_reward_credit": process_credit,
        "outcome_only_can_distinguish_steps": len(set(outcome_credit)) > 1,
        "process_reward_can_distinguish_steps": len(set(process_credit)) > 1,
    }

A real run — a toy trajectory, both credit-assignment methods, run on the identical steps

Input — a 3-step toy trajectory where only the middle step was actually the mistake: step 1 (correct sub-action), step 2 (the actual mistake), step 3 (correct sub-action). The trajectory failed overall, because of step 2 — final outcome reward: -1.

Output — outcome-only credit, the single final reward broadcast across all three steps: [-1, -1, -1]. outcome_only_can_distinguish_steps: false — this credit assignment is bitwise identical to what a trajectory where all three steps were bad would have produced. There's no way to look at this output and recover which step actually caused the failure.

Output — process-level credit, each step scored on its own: [1, -1, 1]. process_reward_can_distinguish_steps: true — the actual culprit is visible directly in the numbers, not just asserted.

This is the concrete version of the credit assignment problem: reinforcement learning needs to know which action in a trajectory deserves credit or blame, and a single scalar at the end of a multi-step trajectory structurally cannot answer that question on its own, no matter how well-tuned the final reward is. A 2026 survey of the field counts 69 papers (56 core methods) trying to close exactly this gap — from GRPO's group-relative baseline to methods like GiGPO's "anchor state grouping," which groups identical states that recur across different rollouts to get finer-grained, step-level credit while keeping GRPO's critic-free, low-memory properties.

DPO: preference pairs, no separate reward model

Classic RLHF is a two-stage pipeline: train a separate reward model on human preference data, then run PPO against it. Direct Preference Optimization collapses this into one step — the paper's own framing: the approach "enable[s] extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss." No reward model, no RL sampling loop during training — just a loss computed directly over (prompt, chosen, rejected) triples. The reported trade-off is favorable, not just simpler: DPO is described as "stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning," while "exceed[ing] PPO-based RLHF in ability to control sentiment of generations" and matching or improving quality in summarization and dialogue.

def _text(response, client, model):
    # client.messages.create(...) -- see agent_loop.py for the full call shape.
    ...


import json

_PROMPT = "Write a one-sentence tagline for a fictional productivity app called Nimbus."

_JUDGE_INSTRUCTION = (
    "You are judging two taglines for the same fictional product, for clarity and how "
    "specific/concrete (vs generic) they are. Reply with strict JSON only: "
    '{"better": "A" or "B", "reason": "one sentence"}'
)


def dpo_pair_demo(client, model) -> dict:
    # No explicit temperature -- default sampling already produces real
    # variance across independent calls.
    response_a = client.messages.create(model=model, max_tokens=100, messages=[{"role": "user", "content": _PROMPT}])
    response_b = client.messages.create(model=model, max_tokens=100, messages=[{"role": "user", "content": _PROMPT}])
    completion_a, completion_b = _text(response_a, client, model).strip(), _text(response_b, client, model).strip()

    judge_prompt = f"{_JUDGE_INSTRUCTION}\n\nA: {completion_a}\nB: {completion_b}"
    judge_response = client.messages.create(model=model, max_tokens=150, messages=[{"role": "user", "content": judge_prompt}])
    judge_text = _text(judge_response, client, model)
    verdict = json.loads(judge_text[judge_text.index("{"):judge_text.rindex("}") + 1])

    chosen, rejected = (completion_a, completion_b) if verdict["better"] == "A" else (completion_b, completion_a)
    return {
        "completion_a": completion_a, "completion_b": completion_b, "judge_verdict": verdict,
        "dpo_pair": {"prompt": _PROMPT, "chosen": chosen, "rejected": rejected},
    }

A real run — two real completions, a real judge verdict, a real preference pair

Input, identical for both completions: "Write a one-sentence tagline for a fictional productivity app called Nimbus." — no temperature was set explicitly (this SDK version's Messages API has no top-level temperature parameter at all), and default sampling still produced two genuinely different completions on two independent calls.

Output — completion A: "Nimbus: Your thoughts organized, your tasks flowing, your potential limitless." Output — completion B: "Nimbus: Rise above the chaos and let your tasks drift into focus."

Output — the real judge verdict: "B is more concrete and specific with vivid metaphors ('drift into focus', 'rise above chaos') that better differentiate the product, while A relies on generic aspirational language that could apply to almost any productivity tool."

The resulting DPO pair, constructed directly from that verdict: chosen = completion B, rejected = completion A. This is the exact shape DPO's loss consumes — no reward model was trained to produce it, just one comparison call over two real samples.

Interview angle

Weak answer to "how would you train an agent to be more reliable at a multi-step task": "Just run RL with a reward for task success." This names the reward signal and skips the actual hard part this page's own real run demonstrates twice: a single outcome reward can't tell which step in a multi-step trajectory deserves the blame, and (separately) a naive group-relative baseline silently contributes zero gradient on every prompt the model already handles perfectly or fails completely — which, on a real model partway through training, is not a rare edge case.

Strong answer: outcome reward and process reward solve different problems and have different costs. Outcome reward is cheap to compute (often just a verifier) but structurally can't do credit assignment across steps — this page's own toy trajectory shows the exact failure: identical credit for a uniformly-bad run and a run where only one step failed. Process reward fixes that but costs real labeling or estimation effort (PRM800K needed 800,000 human labels; MC-estimated alternatives need consensus filtering to be trustworthy at all). Separately, whichever reward you pick, the optimizer itself needs real variance to learn from — GRPO's group-relative baseline goes to exactly zero gradient on a zero-variance group, which is why DAPO's dynamic sampling (filter out accuracy-0-or-1 groups, keep sampling until you find real variance) is a documented, measured fix, not a hypothetical one.

Follow-up to expect: "when would you reach for DPO instead of a full RL loop like GRPO?" DPO's real advantage is that it needs only static preference pairs, not a live sampling loop against an evolving policy — cheaper and more stable, which is exactly why the paper reports it matching or beating PPO-based RLHF while eliminating "the need for sampling from the LM during fine-tuning." GRPO (and its multi-turn agentic descendants) earn their extra complexity specifically when the reward signal itself needs to come from actually running the policy — executing a multi-step trajectory, checking whether a tool call succeeded, verifying an intermediate state — which a static, pre-collected preference dataset structurally cannot capture.

Build it yourself — 30 minutes

  1. Implement grpo_advantages exactly as the formula reads: group mean, group standard deviation, (reward - mean) / std. Test it against a group with real variance, then deliberately test it against a group where every reward is identical — read the actual numbers, don't just assume what happens.
  2. Build the dynamic-sampling filter: given several candidate groups, keep only the ones with accuracy strictly between 0 and 1. Feed it a sequence where the first few groups are degenerate and a later one isn't, and verify it actually skips the useless ones rather than training on them.
  3. Construct a toy multi-step trajectory (3-5 steps) with a clear ground-truth "which step was actually wrong." Implement outcome-only credit assignment (broadcast the final reward to every step) and process-level credit (score each step individually), and check programmatically whether each method's output can actually distinguish the bad step from the good ones — don't just eyeball it.
  4. Generate two real completions to the same prompt and have a real judge call rank them into a (chosen, rejected) pair. Read both completions and the judge's stated reason before trusting the pair — a judge call can fail to parse or produce a verdict that doesn't hold up, the same way any other LLM call can.

Scenario Check

Sources & further reading