Skip to content

How to score your agent's behavior with eval

Murmur's eval system lets you run your capsule against a dataset of test cases and automatically score each session. Scores are written to eval.jsonl alongside trace.jsonl in each session workdir and can be compared across runs with mur eval diff. This guide shows how to wire up murmur-hook-eval, write a dataset, run it, and interpret the results.

The relevant manifest options are:

Option Controls
observability.eval.scorers List of scorer configurations; at least one required to activate the hook
observability.eval.dataset_id Label applied to every dataset_run record in eval.jsonl

Step 1 — declare murmur-hook-eval in the manifest

murmur-hook-eval is a hook artifact — it runs at fixed lifecycle points and writes scores. Create a murmur.yaml file with runtime: hook and configure scorers under observability.eval:

name: my-agent
version: "0.1.0"

artifacts:
  - name: murmur-driver-anthropic
    version: "1.0.0"
    runtime: driver
  - name: murmur-hook-eval
    version: "1.0.0"
    runtime: hook
    capabilities:
      filesystem:
        scope: .          # the hook writes the session's eval.jsonl scores here; hooks get no filesystem by default
  - name: murmur-tool-editor
    version: "1.0.0"
    runtime: tool

capabilities:
  network:
    allow:
      - https://api.anthropic.com
  filesystem:
    allow:
      - .

inference:
  transport: http
  endpoint: https://api.anthropic.com
  model: claude-sonnet-5
  api_key: ${ANTHROPIC_API_KEY}
  driver:
    artifact: murmur-driver-anthropic

observability:
  eval:
    dataset_id: my-dataset
    scorers:
      - type: exit_ok
        name: success_check
      - type: max_turns
        name: turn_limit
        max: 5
      - type: max_tokens
        name: response_size
        max: 3000
      - type: tool_sequence
        name: tool_order
        expected: [murmur-tool-editor]

name: my-agent
version: "0.1.0"

artifacts:
  - name: murmur-driver-openai
    version: "1.0.0"
    runtime: driver
  - name: murmur-hook-eval
    version: "1.0.0"
    runtime: hook
    capabilities:
      filesystem:
        scope: .          # the hook writes the session's eval.jsonl scores here; hooks get no filesystem by default
  - name: murmur-tool-editor
    version: "1.0.0"
    runtime: tool

capabilities:
  network:
    allow:
      - https://api.openai.com
  filesystem:
    allow:
      - .

inference:
  transport: http
  endpoint: https://api.openai.com
  model: o3-mini-high
  api_key: ${OPENAI_API_KEY}
  driver:
    artifact: murmur-driver-openai

observability:
  eval:
    dataset_id: my-dataset
    scorers:
      - type: exit_ok
        name: success_check
      - type: max_turns
        name: turn_limit
        max: 5
      - type: max_tokens
        name: response_size
        max: 3000
      - type: tool_sequence
        name: tool_order
        expected: [murmur-tool-editor]

name: my-agent
version: "0.1.0"

artifacts:
  - name: murmur-driver-deepseek
    version: "1.0.0"
    runtime: driver
  - name: murmur-hook-eval
    version: "1.0.0"
    runtime: hook
    capabilities:
      filesystem:
        scope: .          # the hook writes the session's eval.jsonl scores here; hooks get no filesystem by default
  - name: murmur-tool-editor
    version: "1.0.0"
    runtime: tool

capabilities:
  network:
    allow:
      - https://api.deepseek.com
  filesystem:
    allow:
      - .

inference:
  transport: http
  endpoint: https://api.deepseek.com
  model: deepseek-r1
  api_key: ${DEEPSEEK_API_KEY}
  driver:
    artifact: murmur-driver-deepseek

observability:
  eval:
    dataset_id: my-dataset
    scorers:
      - type: exit_ok
        name: success_check
      - type: max_turns
        name: turn_limit
        max: 5
      - type: max_tokens
        name: response_size
        max: 3000
      - type: tool_sequence
        name: tool_order
        expected: [murmur-tool-editor]

What each scorer does:

Scorer type Passes when
exit_ok The session exited with status: ok (not failed or max_turns_reached)
max_turns The session used at most max inference turns
max_tokens Total input + output tokens did not exceed max
tool_sequence The expected list is a subsequence of the observed tool calls (same order, gaps allowed)

The name field for each scorer is the key that appears in eval.jsonl score records.

Step 2 — install dependencies

mur install
Different ways to install artifacts

mur install needs to know where to fetch artifacts from. You have two options:

Option A — configure a registry source in ~/.murmur/config.yaml:

registry:
  default: official
  sources:
    - name: official
      type: github
      repo: <owner>/<repo>
      token: "${GITHUB_TOKEN}"

Then install by artifact name and version:

mur install <artifact-name@version>

Option B — pass a full GitHub reference and skip configuration entirely:

mur install github:<username>/<repo>@<tag>

See Installing artifacts to learn more.

Step 3 — write a dataset

A dataset is a JSONL file — one JSON object per line, one line per test case. The conventional location is eval.jsonl in the project directory — the same folder as murmur.yaml, where mur eval run looks by default.

Create one task file in the project root:

task.md: 

Read MURMUR.md, then draft a one-sentence summary of the Murmur project. Reply with only the final sentence.

The task starts with a tool call — the agent reads MURMUR.md (the capsule environment reference injected into every workdir) before answering. This is important: tool_sequence applies to every case in the dataset, so every case must call the tool. Grounding the task in a file read makes that natural.

The same task runs for all four cases — what differs is the case_id, which names the scorer each case is primarily meant to verify.

Then create eval.jsonl:

{ "case_id": "exit-check",  "task_path": "task.md" }
{ "case_id": "turns-check", "task_path": "task.md" }
{ "case_id": "size-check",  "task_path": "task.md" }
{ "case_id": "tool-check",  "task_path": "task.md" }

Each task_path points to a file that is copied into the capsule workdir as task.md before the session starts. exit-check verifies success_check (clean exit); turns-check verifies turn_limit (read + answer takes two inference turns, well within five); size-check verifies response_size (MURMUR.md content adds roughly 1200 tokens to the second turn, bringing the total to ~2450, within the 3000-token budget); tool-check verifies tool_order (the editor was called before the answer).

The case_id is injected into the hook environment as MURMUR_CASE_ID and appears in every dataset_run record — it's how you identify each case in the output.

Step 4 — run the dataset

Run from the directory containing murmur.yaml:

mur eval run

Both arguments default: capsule to the current directory and dataset to eval.jsonl. To override either:

mur eval run ./other-capsule                                # explicit capsule
mur eval run --dataset ./other.jsonl                        # explicit dataset
mur eval run ./other-capsule --dataset ./other.jsonl        # both explicit

mur eval run runs one capsule session per case, sequentially. For each case it:

  1. Stages the session — allocates a per-session workdir (workdir/ses_xxx/) and loads artifacts
  2. Copies task_path into the session workdir as task.md
  3. Launches the session
  4. Reads eval.jsonl from the session workdir (written by the hook at session end)
  5. Prints a result line

Output as cases complete:

Running 4 case(s) …
  case: exit-check
    result: pass  session: ses_6801f81dd28b4a9daf434e8324c4793e
  case: turns-check
    result: pass  session: ses_7902a92ee39c5baebf545f9435d5804f
  case: size-check
    result: pass  session: ses_9b14c03df47e6a1c84523f7896d5920f
  case: tool-check
    result: pass  session: ses_ab25d04ef58f7643bce9fbd67da681c3

After all cases:

── Summary ──────────────────────────────────────
pass: 4/4

  exit-check    pass  response_size=1.00 success_check=1.00 tool_order=1.00 turn_limit=1.00  (/path/to/workdir/ses_6801...)
  turns-check   pass  response_size=1.00 success_check=1.00 tool_order=1.00 turn_limit=1.00  (/path/to/workdir/ses_7902...)
  size-check    pass  response_size=1.00 success_check=1.00 tool_order=1.00 turn_limit=1.00  (/path/to/workdir/ses_9b14...)
  tool-check    pass  response_size=1.00 success_check=1.00 tool_order=1.00 turn_limit=1.00  (/path/to/workdir/ses_ab25...)

All four cases pass every scorer. Reading MURMUR.md takes one inference turn (the tool call) and answering takes a second, so turn_limit sees two turns. The tool result injects MURMUR.md's content into the second turn's context, which pushes total tokens to around 2450 — that's why max_tokens is set to 3000 rather than something tighter.

Step 5 — read the eval output

Each case produces its own eval.jsonl in the session workdir. Inspect the most recent session, or any session by suffix:

mur eval show               # most recent session
mur eval show 6801          # session whose ID ends in "6801"

Output:

── Eval: workdir/ses_6801.../eval.jsonl ─────────────────────────────────────

── Scorers ──────────────────────────────────────
  response_size            1/1 pass  (100.0%)
  success_check            1/1 pass  (100.0%)
  tool_order               1/1 pass  (100.0%)
  turn_limit               1/1 pass  (100.0%)

── Overall ──────────────────────────────────────
  result:  pass
  case:    exit-check
  dataset: my-dataset

── Score summary ────────────────────────────────
  response_size            1.0000
  success_check            1.0000
  tool_order               1.0000
  turn_limit               1.0000

── Worst events ─────────────────────────────────
  (no failing events)

The Worst events section lists the individual scoring events that failed — useful for diagnosing which specific turn or tool call tripped a scorer. Here it's empty because all four scorers passed.

For programmatic use:

mur eval show --json

{
  "overall": "pass",
  "scorers": {
    "response_size": { "pass": 1, "fail": 0, "total": 1, "pass_rate": 1.0 },
    "success_check": { "pass": 1, "fail": 0, "total": 1, "pass_rate": 1.0 },
    "turn_limit": { "pass": 1, "fail": 0, "total": 1, "pass_rate": 1.0 }
  },
  "dataset_run": { "overall": "pass", "scores": { "response_size": 1.0, "success_check": 1.0, "turn_limit": 1.0 } }
}

Step 6 — compare two eval runs

After re-running the dataset (for example, after changing the system prompt or swapping the model), compare the new results against a baseline to check for regressions:

mur eval diff 21f7 93a8

Output when both runs pass all scorers:

Scorer                   Run A          Run B          Delta
──────────────────────── ────────────── ────────────── ──────────────────────────
response_size            100.0%         100.0%         =
success_check            100.0%         100.0%         =
tool_order               100.0%         100.0%         =
turn_limit               100.0%         100.0%         =

overall                  pass           pass

If a scorer regressed — for example response_size dropped from 100% to 0% after a prompt change — the delta would show -100.0pp (A better) and overall would change to fail. That is the signal to investigate.

Step 7 — interpret a failing scorer

Each failing scorer tells you something specific:

Scorer fails What it usually means
exit_ok The session errored out or hit the turn cap — check the trace for the exit status
max_turns The agent needed more turns than the budget allows — consider raising max or tightening the system prompt
max_tokens The session used more tokens than the budget — the agent is producing verbose output; try a more constrained prompt or a lower-verbosity model
tool_sequence The agent called tools in the wrong order or skipped a required tool — check the inference events in the trace

When a scorer fails unexpectedly, open the companion trace for the same session:

mur trace show
Different ways to identify a session

mur trace show locates a session three ways:

No argument — most recent session:

mur trace show

Finds the lexicographically largest ses_* entry in workdir/ — always the most recently created session, because session IDs are UUID v7 (time-ordered).

Short suffix — type the last few characters:

mur trace show 3e4b

Matches any session whose ID ends with the given string. Use 4 or more characters to avoid ambiguity. Matching is case-insensitive. If the suffix matches more than one session, mur lists the candidates and asks you to be more specific.

Full session ID:

mur trace show ses_6801f81dd28b4a9daf434e8324c4793e

Resolves the session directly without scanning workdir/.

Legacy file path:

mur trace show path/to/trace.jsonl

Any argument containing / or ending in .jsonl is treated as a literal file path. Kept for backward compatibility.

Use --workdir <path> if your session directory is not ./workdir.

Other trace exploration commands

mur trace has four subcommands for exploring session output:

mur trace show — print the full trace for a session to the terminal. Accepts a session ID, suffix, or no argument (most recent session).

mur trace steps — show a turn-by-turn summary of what the agent did in a session. Pass --verbose to include a truncated summary of each tool's input.

mur trace diff — compare the traces of two sessions side by side. Useful for spotting behavioural regressions between runs.

mur trace report — generate a structured summary report from a session's trace. Covers token usage, tool calls, latency, and other session-level metrics.

Summary

Step What you do
Declare murmur-hook-eval with runtime: hook Activates the eval hook for this capsule
Grant it capabilities.filesystem.scope on its own entry Lets it write eval.jsonl; hooks have no filesystem access by default (see Hook capabilities)
Declare murmur-tool-editor with runtime: tool Gives the agent file-read capability; required for tool_sequence cases
Configure observability.eval.scorers Defines the scoring criteria — all scorers run for every case
Write eval.jsonl One case per line, each with a case_id and task_path
mur eval run Runs all cases against eval.jsonl and prints a pass/fail summary
mur eval show <eval.jsonl> Shows which scorers passed, which failed, and why
mur eval diff <a> <b> Compares two runs to check for regressions