Skip to content

Data and Environments

Hands-on

Full runnable recipe: data-environments/ in the companion cookbook — a real, complete SWE-smith-style task-synthesis-and-verification loop, run start to finish against real code and real tests.

TL;DR — quick revision
  • Real trajectory-synthesis methods turn scarce, expensive real data into cheap, scalable synthetic data — with real, automated verification built in, not just generation. AgentTrek converts web tutorials into agent trajectories at a real, verified cost of "$0.55 per high-quality trajectory without human annotators," using "a VLM-based evaluator [to verify] trajectory correctness." ToolACE builds "a comprehensive API pool of 26,507 diverse APIs" across 390 domains, checked by a real "dual-layer verification system combining rule-based and model-based checks."
  • SWE-Gym and SWE-smith attack the same real bottleneck — realistic SWE training environments — from two different angles. SWE-Gym curates "2,438 real-world Python task instances" from actual GitHub issues, producing "up to 19% absolute gains in resolve rate" on SWE-bench. SWE-smith instead "automatically synthesizes" tasks by breaking tests in "50k instances sourced from 128 GitHub repositories" — no real historical bug report required at all.
  • A real repro of SWE-smith's exact mechanism, run start to finish: a real Claude call introduced a genuine, subtle boundary bug into working code; a real pytest run confirmed the task was real (4 of 5 tests passed, 1 genuinely failed); a second real Claude call, given only the real failing test output, fixed it; a real pytest re-run confirmed all 5 tests passed.
  • "Verifiers" is the real, general term for what makes RL training at scale possible: a deterministic pass/fail signal instead of an LLM's opinion. RLVR's own framing: "for code, a compiler ran the output and returned pass or fail, producing binary rewards: 1 for correct, 0 for wrong." Prime Intellect's real, open-source verifiers library formalizes this as an Environment — "a dataset of task inputs, a harness for the model... and a reward function or rubric to score the model's performance."

Trajectory synthesis: cheap, scalable, and verified

Real agent trajectories — a full sequence of an agent's actions and observations completing a task — are expensive to collect from humans and scarce in the wild. Real, current methods synthesize them instead, with automated verification built directly into the pipeline rather than trusting the generation step alone. AgentTrek's real mechanism: harvest web tutorials, convert them into structured task specs, then "use a visual-language model (VLM) agent to execute these instructions in real environments, while a VLM-based evaluator verifies trajectory correctness" — real, verified cost: "$0.55 per high-quality trajectory without human annotators." APIGen-MT takes a two-phase approach — task blueprints "validated through format/execution checks and an LLM committee review," then turned into full trajectories "through simulated human-agent interplay" — with 5K real trajectories openly released alongside the resulting models.

ToolACE: scale and verification for tool-calling data specifically

The methods above synthesize whole task trajectories. ToolACE narrows the same idea to one specific skill an agent needs constantly: calling the right tool, with the right arguments, in realistic conversation. Instead of scraping real tool-calling transcripts (which are scarce and inconsistent), it generates its own — both the tools being called and the dialogue calling them — then checks the result two different ways rather than trusting either check alone. Concretely, it builds "a comprehensive API pool of 26,507 diverse APIs" across 390 real domains through "a novel self-evolution synthesis process," with dialogs generated "through the interplay among multiple agents, guided by a formalized thinking process." Its real, distinguishing safeguard: a "dual-layer verification system combining rule-based and model-based checks" — not trusting either a deterministic rule or a model's judgment alone. At publication, ToolACE-8B topped the BFCL-v1 leaderboard at 91.41% overall accuracy, ahead of Claude-3.5-Sonnet's 90.53%; worth noting that BFCL has since moved to a harder v3 leaderboard (adding "live" categories), where ToolACE-8B's real score is lower (59.22, rank #3) — a real reminder that a benchmark's own difficulty isn't fixed over time, and a headline number needs its leaderboard version attached.

SWE-Gym and SWE-smith: real tasks vs. synthesized tasks, at real scale

Two real, complementary approaches to the same problem: training data for software-engineering agents needs a real, executable environment, not just text. SWE-Gym curates "2,438 real-world Python task instances," each "comprising a codebase with an executable runtime environment, unit tests, and a task specified in natural language," sourced from actual GitHub issues — real, and "the first publicly available training environment combining real-world SWE tasks from GitHub issues with pre-installed dependencies and executable test verification." Training on it produced real, measured gains: "up to 19% absolute gains in resolve rate on the popular SWE-Bench Verified and Lite test sets," reaching 32.0% and 26.0% respectively with trained verifiers added — a real state-of-the-art for open-weight agents at publication.

SWE-smith takes the opposite real approach: instead of curating existing real bug reports, it "automatically synthesizes... task instances that break existing test(s) in the codebase" from arbitrary working repositories — no historical PR or issue required at all. The real scale gap this unlocks is stark: "50k instances sourced from 128 GitHub repositories," against prior work's "at most 1,000s of training instances from 11 or fewer GitHub repositories." The resulting model, SWE-agent-LM-32B, reached 40.2% Pass@1 on SWE-bench Verified — state of the art among open-source models at publication. This page's own repro reproduces SWE-smith's exact mechanism directly, at a much smaller scale but with the identical underlying idea.

Repro: synthesizing a real task, then verifying a real fix

This repro reproduces SWE-smith's exact mechanism, at a much smaller scale: start from ordinary working code with a passing test suite, have a model introduce one realistic bug into it (no historical bug report needed), then confirm with a real test run that the result is a genuine, narrow, solvable failure rather than either nothing at all or a total collapse. Three pieces: the starting code and its test suite, a verifier that actually executes the tests rather than asking a model to judge correctness, and the prompt that asks a model to synthesize a subtle bug.

_ORIGINAL_CODE = '''\
def merge_intervals(intervals):
    """Merge overlapping intervals. Input: list of [start, end] pairs."""
    if not intervals:
        return []
    intervals = sorted(intervals, key=lambda pair: pair[0])
    merged = [intervals[0]]
    for start, end in intervals[1:]:
        last_start, last_end = merged[-1]
        if start <= last_end:
            merged[-1] = [last_start, max(last_end, end)]
        else:
            merged.append([start, end])
    return merged
'''

_TEST_CODE = '''\
from solution import merge_intervals

def test_no_overlap():
    assert merge_intervals([[1, 2], [3, 4]]) == [[1, 2], [3, 4]]

def test_simple_overlap():
    assert merge_intervals([[1, 3], [2, 6], [8, 10]]) == [[1, 6], [8, 10]]

def test_touching_intervals():
    assert merge_intervals([[1, 4], [4, 5]]) == [[1, 5]]

def test_empty():
    assert merge_intervals([]) == []

def test_unsorted_input():
    assert merge_intervals([[5, 6], [1, 3]]) == [[1, 3], [5, 6]]
'''

import os
import subprocess
import sys
import tempfile


def run_pytest(code: str) -> dict:
    """The real, deterministic verifier -- actually executes the real test suite
    against real code in a fresh temp directory. No LLM judgment involved."""
    with tempfile.TemporaryDirectory() as tmp:
        with open(os.path.join(tmp, "solution.py"), "w") as f:
            f.write(code)
        with open(os.path.join(tmp, "test_solution.py"), "w") as f:
            f.write(_TEST_CODE)
        result = subprocess.run(
            [sys.executable, "-m", "pytest", "test_solution.py", "-v", "--tb=short"],
            cwd=tmp, capture_output=True, text=True, timeout=30,
        )
    return {"passed": result.returncode == 0, "output": result.stdout[-1500:]}

_SYNTHESIZE_PROMPT_TEMPLATE = (
    "Here is a working Python function:\n\n"
    "```python\n{original_code}```\n\n"
    "Introduce exactly ONE subtle, realistic bug into this function -- the kind a "
    "developer might genuinely introduce by accident (an off-by-one, a wrong comparison "
    "operator, a mishandled edge case). Do not add comments pointing out the bug. "
    "Return ONLY the complete modified function in a python code block."
)

Put together: a real, working merge_intervals function with a real, passing 5-test suite. A real Claude call introduces exactly one subtle bug — SWE-smith's own approach, generating a task from arbitrary working code rather than requiring an existing bug report. The real verifier above is the same shape RLVR and SWE-Gym both rely on: actually run the code, don't ask an LLM whether it looks right.

A real run, unforced — full synthesis-then-fix loop

Synthesis: Claude changed one comparison operator, start <= last_end to start < last_end — a genuinely subtle boundary bug (intervals that exactly touch, like [1,4] and [4,5], stop merging). Real verification: 4 passed, 1 failed — specifically test_touching_intervals, exactly the case the injected bug breaks, nothing else. A real, well-formed synthesized task: hard enough to be genuine, narrow enough to be solvable.

Fix: given only the real failing pytest output — not the original correct code — a second Claude call correctly diagnosed the boundary condition and restored start <= last_end. Real re-verification: 5 passed — the fix genuinely worked, confirmed by execution, not by the fixing model's own confidence.

A real bug was caught and fixed in this recipe's own code along the way: an earlier version indexed response.content[0].text directly and broke with AttributeError: 'ThinkingBlock' object has no attribute 'text' — the first content block isn't reliably the text block. Fixed by filtering for block.type == "text" explicitly, then this clean run was captured.

Verifiers: the real mechanism that makes any of this trainable at scale

Every method on this page depends on the same real, underlying requirement: a way to score a trajectory or an output without a human in the loop, cheaply and reliably enough to run millions of times. RLVR's own framing names the pattern directly: "for code, a compiler ran the output and returned pass or fail, producing binary rewards: 1 for correct, 0 for wrong" — the same real mechanism this page's own repro just ran, by hand, on one example. Prime Intellect's real, open-source verifiers library formalizes the idea as a reusable unit: an Environment is "a dataset of task inputs, a harness for the model (tools, sandboxes, context management, etc.), and a reward function or rubric to score the model's performance," with a real, stated reason for decoupling verification logic from any specific training stack: "Environment implementations are often tied to a specific training RL stack and can be difficult to adapt to a new trainer."

What this means in practice

The real throughline across every method on this page: data and environments for training agents don't come from finding more real examples, they come from designing a real, automated way to generate and check examples at scale. SWE-Gym curates real tasks with executable verification already attached; SWE-smith and AgentTrek synthesize tasks and verify them automatically instead of curating; ToolACE layers rule-based and model-based checks on generated dialogues; the verifiers library packages the whole pattern — dataset, harness, reward function — as a reusable unit. This page's own repro is the smallest possible version of the same idea: one real bug synthesized, one real test run to confirm it's genuine, one real fix attempted, one real test run to confirm it worked. Nothing here depended on an LLM's opinion about correctness — every real verdict came from actually executing code.

Interview angle

Weak answer to "how would you get training data for a coding agent?": "Scrape real GitHub issues and PRs." SWE-smith's own real, verified numbers are a direct counterexample to treating that as the only real lever — 50k instances from 128 repos, generated by breaking tests in arbitrary working code, dwarfs what curating real historical bug reports alone can practically produce ("at most 1,000s... from 11 or fewer repositories").

Strong answer: name the real, general pattern underneath every method on this page — synthesize the task, then verify it's genuine and solvable with a deterministic check, not an LLM's judgment. This page's own repro demonstrates the mechanism concretely: real pytest execution, not a model saying "yes, this looks like a real bug" or "yes, this looks fixed." Then connect it to why this matters for RL specifically: RLVR's binary pass/fail reward only scales because the verifier is cheap and automated — the same real property SWE-Gym, SWE-smith, and ToolACE's own verification layers are all built around.

Follow-up to expect: "what happens when you can't write a clean pass/fail verifier for a task?" A real, honest answer grounded in this page's own material: that's exactly the gap ToolACE's dual-layer verification (rule-based plus model-based) and AgentTrek's VLM-based evaluator are built to cover — when a deterministic check isn't available, a model-based verifier is the real fallback, at the real cost of being less reliable than actually running code, which is why the strongest pipelines on this page layer both rather than picking one.

Build it yourself — 30 minutes

  1. Pick a real, small function you have with a real, passing test suite (or write one — 4-5 tests is enough).
  2. Have a real LLM call introduce one subtle bug into it, then actually run the test suite against the result — this page's own repro predicts a clean, informative partial failure (some tests pass, the ones testing the broken behavior fail), not a total collapse.
  3. Feed only the real failing test output (not the original code) to a second LLM call and ask it to fix the bug. Re-run the real test suite to verify.
  4. If you're building anything that needs to score model outputs at scale, check whether a deterministic verifier (execution, exact match) is possible before reaching for an LLM-as-judge — this page's own material argues that's the more reliable, and dramatically cheaper, default whenever the task allows it.

Scenario Check

Sources & further reading