How to score your agent's behavior with eval

Murmur's eval system lets you run your capsule against a dataset of test cases and automatically score each session. Scores are written to hooks/eval.jsonl beside trace.jsonl in each session's records directory, where the capsule's own commands cannot change them, and can be compared across runs with mur eval diff. This guide shows how to wire up murmur/hook-eval, write a dataset, run it, and interpret the results.

The relevant manifest options are:

Option	Controls
observability.eval.scorers	List of scorer configurations; at least one required to activate the hook
observability.eval.dataset_id	Label applied to every dataset_run record in eval.jsonl

---

Step 1 — declare murmur/hook-eval in the manifest

murmur/hook-eval is a hook artifact — it runs at fixed lifecycle points and writes scores. Create a murmur.yaml file with runtime: hook and configure scorers under observability.eval:

AnthropicOpenAIDeepSeek

name: my-agent
version: "0.1.0"

artifacts:
  - name: murmur/driver-anthropic
    version: "0.19.0"
    runtime: driver
    gateway:
      endpoint: https://api.anthropic.com
      api_key: ${ANTHROPIC_API_KEY}
  - name: murmur/hook-eval
    version: "0.11.0"
    runtime: hook
    capabilities:
      filesystem:
        records: true     # the hook writes the session's eval.jsonl beside its trace; hooks get no filesystem by default
  - name: murmur/tool-read
    version: "0.2.0"
    runtime: tool

capabilities:
  filesystem:
    allow:
      - .

inference:
  transport: http
  model: claude-sonnet-5
  driver:
    artifact: murmur/driver-anthropic

observability:
  eval:
    dataset_id: my-dataset
    scorers:
      - type: exit_ok
        name: success_check
      - type: max_turns
        name: turn_limit
        max: 5
      - type: max_tokens
        name: response_size
        max: 3000
      - type: tool_sequence
        name: tool_order
        expected: [tool-read]

name: my-agent
version: "0.1.0"

artifacts:
  - name: murmur/driver-openai
    version: "0.19.0"
    runtime: driver
    gateway:
      endpoint: https://api.openai.com/v1
      api_key: ${OPENAI_API_KEY}
  - name: murmur/hook-eval
    version: "0.11.0"
    runtime: hook
    capabilities:
      filesystem:
        records: true     # the hook writes the session's eval.jsonl beside its trace; hooks get no filesystem by default
  - name: murmur/tool-read
    version: "0.2.0"
    runtime: tool

capabilities:
  filesystem:
    allow:
      - .

inference:
  transport: http
  model: o3-mini-high
  driver:
    artifact: murmur/driver-openai

observability:
  eval:
    dataset_id: my-dataset
    scorers:
      - type: exit_ok
        name: success_check
      - type: max_turns
        name: turn_limit
        max: 5
      - type: max_tokens
        name: response_size
        max: 3000
      - type: tool_sequence
        name: tool_order
        expected: [tool-read]

name: my-agent
version: "0.1.0"

artifacts:
  - name: murmur/driver-deepseek
    version: "0.17.0"
    runtime: driver
    gateway:
      endpoint: https://api.deepseek.com
      api_key: ${DEEPSEEK_API_KEY}
  - name: murmur/hook-eval
    version: "0.11.0"
    runtime: hook
    capabilities:
      filesystem:
        records: true     # the hook writes the session's eval.jsonl beside its trace; hooks get no filesystem by default
  - name: murmur/tool-read
    version: "0.2.0"
    runtime: tool

capabilities:
  filesystem:
    allow:
      - .

inference:
  transport: http
  model: deepseek-r1
  driver:
    artifact: murmur/driver-deepseek

observability:
  eval:
    dataset_id: my-dataset
    scorers:
      - type: exit_ok
        name: success_check
      - type: max_turns
        name: turn_limit
        max: 5
      - type: max_tokens
        name: response_size
        max: 3000
      - type: tool_sequence
        name: tool_order
        expected: [tool-read]

What each scorer does:

Scorer type	Passes when
exit_ok	The session ended with status: ok: its task completed, rather than failing, spending inference.max_turns, hitting a spend ceiling or being canceled. See Status and exit code
max_turns	The session used at most max inference turns
max_tokens	Total input + output tokens did not exceed max
tool_sequence	The expected list is a subsequence of the observed tool calls (same order, gaps allowed)

The name field for each scorer is the key that appears in eval.jsonl score records.

---

Step 2 — install dependencies

mur install

Different ways to install artifacts

mur install fetches an artifact from the source of its scope. murmur/ artifacts come from the built-in source, murmur-nexus/default-artifacts, with no configuration.

Option A — name the source of a scope in ~/.murmur/config.yaml:

registry:
  sources:
    - repo: <owner>/<repo>
      token: "${GITHUB_TOKEN}"

Then install by full name, with or without a version:

mur install <owner>/<artifact-name>@<version>

Option B — pass a full GitHub reference and skip configuration entirely:

mur install github:<owner>/<repo>@<tag>

See Installing artifacts to learn more.

---

Step 3 — write a dataset

A dataset is a JSONL file — one JSON object per line, one line per test case. The conventional location is eval.jsonl in the project directory — the same folder as murmur.yaml, where mur eval run looks by default.

Create one task file in the project root:

task.md:

Read murmur-runtime/MURMUR.md, then draft a one-sentence summary of the Murmur project. Reply with only the final sentence.

The task starts with a tool call — the agent reads MURMUR.md (the capsule environment reference the runtime writes for every agent session, which tool-read reads at murmur-runtime/MURMUR.md) before answering. This is important: tool_sequence applies to every case in the dataset, so every case must call the tool. Grounding the task in a file read makes that natural.

The same task runs for all four cases — what differs is the case_id, which names the scorer each case is primarily meant to verify.

Then create eval.jsonl:

{ "case_id": "exit-check",  "task_path": "task.md" }
{ "case_id": "turns-check", "task_path": "task.md" }
{ "case_id": "size-check",  "task_path": "task.md" }
{ "case_id": "tool-check",  "task_path": "task.md" }

Each task_path points to a file that is copied to the session's task.md before the session starts. exit-check verifies success_check (clean exit); turns-check verifies turn_limit (read + answer takes two inference turns, well within five); size-check verifies response_size (MURMUR.md content adds roughly 1200 tokens to the second turn, bringing the total to ~2450, within the 3000-token budget); tool-check verifies tool_order (tool-read was called before the answer).

The case_id is injected into the hook environment as MURMUR_CASE_ID and appears in every dataset_run record — it's how you identify each case in the output.

---

Step 4 — run the dataset

Run from the directory containing murmur.yaml:

mur eval run

Both arguments default: capsule to the current directory and dataset to eval.jsonl. To override either:

mur eval run ./other-capsule                                # explicit capsule
mur eval run --dataset ./other.jsonl                        # explicit dataset
mur eval run ./other-capsule --dataset ./other.jsonl        # both explicit

mur eval run runs one capsule session per case, sequentially. For each case it:

1. Stages the session — allocates a per-session work folder (workdir/ses_xxx/) and records directory, and loads artifacts
2. Copies task_path to task.md in the session's runtime directory
3. Launches the session
4. Reads hooks/eval.jsonl from the session's records directory (written by the hook at session end). A case whose hook wrote none — one declared scope: . instead of records: true — reports no_scores and a warning naming the grant
5. Prints a result line

Output as cases complete:

Running 4 case(s) …
  case: exit-check
    result: pass  session: ses_6801f81dd28b4a9daf434e8324c4793e
  case: turns-check
    result: pass  session: ses_7902a92ee39c5baebf545f9435d5804f
  case: size-check
    result: pass  session: ses_9b14c03df47e6a1c84523f7896d5920f
  case: tool-check
    result: pass  session: ses_ab25d04ef58f7643bce9fbd67da681c3

After all cases:

── Summary ──────────────────────────────────────
pass: 4/4

  exit-check    pass  response_size=1.00 success_check=1.00 tool_order=1.00 turn_limit=1.00  (/path/to/workdir/ses_6801...)
  turns-check   pass  response_size=1.00 success_check=1.00 tool_order=1.00 turn_limit=1.00  (/path/to/workdir/ses_7902...)
  size-check    pass  response_size=1.00 success_check=1.00 tool_order=1.00 turn_limit=1.00  (/path/to/workdir/ses_9b14...)
  tool-check    pass  response_size=1.00 success_check=1.00 tool_order=1.00 turn_limit=1.00  (/path/to/workdir/ses_ab25...)

It then prints the run's comparison table, the one Step 8 reads, and the run's id, run: evr_….

All four cases pass every scorer. Reading MURMUR.md takes one inference turn (the tool call) and answering takes a second, so turn_limit sees two turns. The tool result injects MURMUR.md's content into the second turn's context, which pushes total tokens to around 2450 — that's why max_tokens is set to 3000 rather than something tighter.

---

Step 5 — read the eval output

Each case produces its own hooks/eval.jsonl in the session's records directory. Inspect the most recent session, or any session by suffix:

mur eval show               # most recent session
mur eval show 6801          # session whose ID ends in "6801"

Output:

── Eval: /home/me/.murmur/records/home/me/eval-capsule/workdir/ses_6801.../hooks/eval.jsonl ──────

── Scorers ──────────────────────────────────────
  response_size            1/1 pass  (100.0%)
  success_check            1/1 pass  (100.0%)
  tool_order               1/1 pass  (100.0%)
  turn_limit               1/1 pass  (100.0%)

── Overall ──────────────────────────────────────
  result:  pass
  case:    exit-check
  dataset: my-dataset

── Score summary ────────────────────────────────
  response_size            1.0000
  success_check            1.0000
  tool_order               1.0000
  turn_limit               1.0000

── Worst events ─────────────────────────────────
  (no failing events)

The Worst events section lists the individual scoring events that failed — useful for diagnosing which specific turn or tool call tripped a scorer. Here it's empty because all four scorers passed.

For programmatic use:

mur eval show --json

{
  "overall": "pass",
  "scorers": {
    "response_size": { "pass": 1, "fail": 0, "total": 1, "pass_rate": 1.0 },
    "success_check": { "pass": 1, "fail": 0, "total": 1, "pass_rate": 1.0 },
    "turn_limit": { "pass": 1, "fail": 0, "total": 1, "pass_rate": 1.0 }
  },
  "dataset_run": { "overall": "pass", "scores": { "response_size": 1.0, "success_check": 1.0, "turn_limit": 1.0 } }
}

---

Step 6 — compare two eval runs

After re-running the dataset (for example, after changing the system prompt or swapping the model), compare the new results against a baseline to check for regressions:

mur eval diff 21f7 93a8

Output when both runs pass all scorers:

Scorer                   Run A          Run B          Delta
──────────────────────── ────────────── ────────────── ──────────────────────────
response_size            100.0%         100.0%         =
success_check            100.0%         100.0%         =
tool_order               100.0%         100.0%         =
turn_limit               100.0%         100.0%         =

overall                  pass           pass

If a scorer regressed — for example response_size dropped from 100% to 0% after a prompt change — the delta would show -100.0pp (A better) and overall would change to fail. That is the signal to investigate.

---

Step 7 — interpret a failing scorer

Each failing scorer tells you something specific:

Scorer fails	What it usually means
exit_ok	The session errored out or hit the turn cap — check the trace for the exit status
max_turns	The agent needed more turns than the budget allows — consider raising max or tightening the system prompt
max_tokens	The session used more tokens than the budget — the agent is producing verbose output; try a more constrained prompt or a lower-verbosity model
tool_sequence	The agent called tools in the wrong order or skipped a required tool — check the inference events in the trace

When a scorer fails unexpectedly, open the companion trace for the same session:

mur trace show

Different ways to identify a session

mur trace show with no argument reads the most recent session:

mur trace show

To name another one, pass an ordinal counting back from the newest (@2), the last 4 or more characters of its ID (3e4b), the full ID, or a path to its trace.jsonl:

mur trace show @2
mur trace show 3e4b
mur trace show ses_6801f81dd28b4a9daf434e8324c4793e
mur trace show path/to/trace.jsonl

Use --workdir <path> if your session directories are not under ./workdir. Every command that names a session takes the same addresses — see Session addresses.

Other trace exploration commands

mur trace has four subcommands for exploring session output:

mur trace show — print the full trace for a session to the terminal.

mur trace steps — show a turn-by-turn summary of what the agent did in a session. Pass --verbose to include a truncated summary of each tool's input.

mur trace diff — compare the traces of two sessions side by side, or with no arguments the two most recent. Useful for spotting behavioural regressions between runs.

mur trace report — generate a structured summary report from a session's trace. Covers token usage, tool calls, latency, and other session-level metrics.

---

Step 8 — compare a harness change by whether the work passes

Scorers judge how a session behaved. To judge whether a change to the harness — a skill, a hook, the system prompt — makes the agent's work pass, give each case a start folder and a check command, and run the changed manifest as a variant against the same starts.

Copy the capsule directory and make the change in the copy:

cp -r . ../variant
$EDITOR ../variant/murmur.yaml

Both manifests must name the same inference.model, and the check's binary must be in both manifests' capabilities.shell.allow: the check runs under the session's own containment.

Give a case a start, a folder copied into each session's work folder before it starts, and a check, a command run in that work folder after the session ends:

eval.jsonl:

{ "case_id": "fix-parser", "task_path": "tasks/fix-parser.md", "start": "cases/parser", "check": "make test" }
{ "case_id": "fix-config", "task_path": "tasks/fix-config.md", "start": "cases/config", "check": { "command": "bash check.sh", "files": "checks/config", "timeout_secs": 120 } }

check.files is copied into the work folder just before the command runs, so a test the agent must not see stays out of the session. The command, its files and its output never reach the agent, the provider or the trace. Exit 0 is pass.

Run every case three times on each manifest:

mur eval run --variant ../variant --repeat 3

Each run starts from a fresh copy of the case's start, checked against the source's digest before launch, with the same task. Repeats alternate which manifest goes first. The run ends with the comparison and the run's id:

run: evr_01a16d24d4d073c1bed590ba1bf056d3
  hint: mur eval diff --run evr_01a16d24d4d073c1bed590ba1bf056d3

Read the comparison again at any time; --run alone reads the most recent run:

mur eval diff --run

── case: fix-parser ────────────────────────
                       baseline                 variant                  delta (variant − baseline)
  check                1/3 pass                 3/3 pass                 +2
  turns                4 (3–6)                  3 (3–4)                  -1
  input tokens         21840 (18210–30117)      15022 (14873–16540)      -6818
  cached tokens        9120 (0–11002)           8890 (7310–9020)         -230
  cache write tokens   4102 (3980–4570)         4022 (3870–4190)         -80
  output tokens        812 (640–1210)           590 (512–700)            -222
  time                 21.4s (18.0s–33.9s)      15.2s (14.1s–17.3s)      -6.2s
  mur trace diff /home/dev/.murmur/records/…/ses_01a1…/trace.jsonl /home/dev/.murmur/records/…/ses_01a2…/trace.jsonl

The check row says which manifest's work passes: here the variant fixed the parser in all three runs and the baseline in one. Each measure is the median over the repeats with the min–max spread, and the delta is the variant's median minus the baseline's. Run the printed mur trace diff line to see where a pair of sessions diverged. A total block after the cases sums every run.

The comparison is kept as a 0600 record beside the session directories:

ls -l ~/.murmur/records$(pwd)/workdir/evr_*.json

For a program deciding whether to keep a change, mur eval diff --run --json prints the same comparison as one JSON object. See mur eval diff --run and the eval-run record.

---

Summary

Step	What you do
Declare murmur/hook-eval with runtime: hook	Activates the eval hook for this capsule
Grant it capabilities.filesystem.records: true on its own entry	Lets it write eval.jsonl into the records directory beside the trace; hooks have no filesystem access by default (see Hook capabilities)
Declare murmur/tool-read with runtime: tool	Gives the agent file-read capability; required for tool_sequence cases
Configure observability.eval.scorers	Defines the scoring criteria — all scorers run for every case
Write eval.jsonl	One case per line, each with a case_id and task_path
mur eval run	Runs all cases against eval.jsonl and prints a pass/fail summary
mur eval show	Shows which scorers passed, which failed, and why, for the most recent session
mur eval diff	Compares the two most recent runs to check for regressions
Give cases a start and a check	Judges each session by whether its work passes, under the session's own containment
mur eval run --variant ../variant --repeat 3	Runs every case on both manifests from the same start
mur eval diff --run	Prints, per case and in total, check passes, turns, tokens and time for both manifests