Agents and models

Configure the agent, model label, and provider auth profile that Workbench uses for evals.

Cases and grading defines grading criteria. Results shows model and agent outcomes after a run.

Agent source

Set the default agent and optional comparison agents in .workbench/agents.yaml:

default: defaultagents:  default:    adapter: codex    with:      auth: default

Omit model to use the adapter's current default. When you set one, Workbench passes the opaque label through without interpreting provider model families.

Selectors

Commands use manifest defaults unless you pass selectors:

workbench eval runworkbench eval gradeworkbench eval run --agents allworkbench eval grade --agents allworkbench eval run --agents default,strictworkbench eval run --versions all --agents all

Use --agents all for every configured comparison agent. Use a named list for one focused proof or a smaller comparison.

Samples

Increase samples to measure variance:

workbench eval run -n 5workbench eval gradeworkbench skill improve -n 2

grade judges existing output and does not accept samples. Increase samples before publishing when a one-sample improvement proof changes source.

Provider auth

Provider-backed agents require provider setup. Hosted runs also require captured Workbench auth:

workbench login codexworkbench login

Adapter auth and Workbench Cloud auth are separate. Hosted runs require Workbench Cloud login, adapter auth for harness-backed agents, and a Cloud skill that can run hosted operations. The adapter owns any provider-native setup guidance.