How to read your agent's trace and compare two runs

Every Murmur agent session produces a trace.jsonl file that captures the complete record of what happened: every inference call, tool invocation, shell command, and compaction event. This guide shows how to read a single session's trace, spot problems, and compare two runs side by side.

The commands in this guide are read-only — they do not modify any file and do not require a running capsule.

---

Where trace.jsonl lives

After mur run -v, the session workdir is printed to stdout:

murmur: url localhost:52222
session: ses_6801f81dd28b4a9daf434e8324c4793e
workdir: /path/to/workdir/ses_6801f81dd28b4a9daf434e8324c4793e
manifest: my-agent v0.1.0
driver: murmur-driver-anthropic (claude-sonnet-4-6)

The trace lives at <workdir>/trace.jsonl. It is always written — even if the session failed — and cannot be suppressed by the capsule.

---

Step 1 — print a session summary

With no argument, mur trace show picks the most recent session automatically:

mur trace show

Different ways to identify a session

mur trace show with no argument reads the most recent session:

mur trace show

To name another one, pass an ordinal counting back from the newest (@2), the last 4 or more characters of its ID (3e4b), the full ID, or a path to its trace.jsonl:

mur trace show @2
mur trace show 3e4b
mur trace show ses_6801f81dd28b4a9daf434e8324c4793e
mur trace show path/to/trace.jsonl

Use --workdir <path> if your session directories are not under ./workdir. Every command that names a session takes the same addresses — see Session addresses.

Other trace exploration commands

mur trace has four subcommands for exploring session output:

mur trace show — print the full trace for a session to the terminal.

mur trace steps — show a turn-by-turn summary of what the agent did in a session. Pass --verbose to include a truncated summary of each tool's input.

mur trace diff — compare the traces of two sessions side by side, or with no arguments the two most recent. Useful for spotting behavioural regressions between runs.

mur trace report — generate a structured summary report from a session's trace. Covers token usage, tool calls, latency, and other session-level metrics.

Output:

── Session ──────────────────────────────────────
session:    ses_6801f81dd28b4a9daf434e8324c4793e
capsule:    my-agent v0.1.0
model:      claude-sonnet-4-6
status:     ok
duration:   3.2s

── Turns ────────────────────────────────────────
count:      4  (max: 10)

── Tokens ───────────────────────────────────────
input:      5,840  (avg 1460/turn)
output:     612  (avg 153/turn)
total:      6,452
provider:   in 5,720, out 590, cached 4,980, cache write 0

── Wire ─────────────────────────────────────────
turn 0  system bbc5e661e106…  tools f9d35d43770d…  response afb8c1747105…  1 messages
turn 1  system bbc5e661e106…  tools f9d35d43770d…  response 4ed87cafe960…  3 messages
turn 2  system bbc5e661e106…  tools f9d35d43770d…  response 7c02d4f1ae33…  5 messages
turn 3  system bbc5e661e106…  tools f9d35d43770d…  response 9b4e6a2f0d18…  7 messages
bodies:     mur trace show --body system --turn 0

── Tool calls ───────────────────────────────────
count:      3  (3 ok, 0 error)  success 100.0%
latency:    avg 28ms
  turn 0  bash 22ms ✓  {"command":"cargo build --release"}
  turn 1  bash 41ms ✓  {"command":"cargo test --workspace"}
  turn 2  bash 21ms ✓  {"command":"cargo clippy"}

── Shell calls ──────────────────────────────────
count:      2
exit codes: 2 ok
latency:    avg 18ms

── Compaction ───────────────────────────────────
fired:      no

Each section maps to a class of events in trace.jsonl. The turn count and token totals give you the most immediate signal — if a session is taking more turns than expected, the turn count shows it immediately. Under Tool calls, each turn row shows the tool that ran, its duration, a ✓/✗ status icon, and — when the trace recorded one — the tool's input as compact JSON (truncated to 120 characters). The provider: row under Tokens appears whenever the driver reported its own counts, and breaks out cached/cache write tokens so you can see whether the provider's prompt cache actually hit. Wire lists each turn's system-prompt, tool-schema and response hashes — not the bytes themselves, which Step 3 covers.

Two more sections appear only when they have something to report, both placed so you cannot miss them:

• Hook failures — printed immediately after Session, above every other section. One ✗ <hook> <lifecycle event> <arm> row per fault, so a broken hook never sits buried below a wall of tool calls.
• Context — printed when an on-task-start hook seeded the task's conversation: the hook's name, whether it committed or was trimmed, the tokens it proposed against its budget, and the ids of the messages it seeded.

---

Step 2 — spot problems in the output

Too many turns — if count is close to the maximum (set in murmur.yaml, default 10 turns), the agent hit the turn cap. Use mur trace steps --verbose to see what it called each turn and what each tool received:

mur trace steps --verbose

Session ses_6801f81dd28b4a9daf434e8324c4793e  (1 task, 4 turns)

task tsk_7a91c2e0…
  turn 0  tool_call  bash
    tool_call  bash  820ms  ✓  {"command":"cargo build --release"}
  turn 1  tool_call  bash
    tool_call  bash  1.2s  ✓  {"command":"cargo test --workspace"}
  turn 2  tool_call  bash
    tool_call  bash  340ms  ✓  {"command":"cargo clippy"}
  turn 3  end_turn

steps walks the session → task → turn tree: every turn nests under its task, and every tool call, shell command or skill call nests under the turn that made it. Repeated calls to the same tool with similar inputs, at the same nesting level turn after turn, is the most common sign of a retry loop.

Tool errors — if the tool calls line shows errors, the model called a tool that returned a failure. Use mur trace show to see which tool failed and what input triggered it.

Latency spikes — avg tool latency and avg shell latency are per-call averages. mur trace show lists every tool call with its individual duration in the Tool calls section, so you can identify the slow call without inspecting the raw file.

Shell failures — when the agent runs shell commands, mur trace show tracks the exit code for each one:

── Shell calls ──────────────────────────────────
count:      3
exit codes: 2 ok, 1 failed (exit 1)

Non-zero exits are passed back to the model as output, not failures — but if the same exit code repeats across turns, the model is retrying a command that keeps failing.

Compaction that never happened — a declined: row under Compaction means a turn crossed the compaction threshold and the context was left alone, so the rest of the session ran over budget:

── Compaction ───────────────────────────────────
fired:      no
declined:   at turn 4  (198,340 tokens)  no_hook_replacement

no_hook_replacement means the capsule declares no compaction hook, or the one it declares returned nothing — declare one to fix it. unresolved_tool_call means a hook did return a replacement and the runtime discarded it because its tool calls and tool results no longer paired up; that is a defect in the hook.

---

Step 3 — print the exact bytes behind a hash

mur trace show never prints a request or response body itself — only its hash, under Wire. To read what a turn actually sent or received, pass --body with a selector and the turn it belongs to:

mur trace show --body system --turn 1

Printing a body requires the session to have recorded one — set trace.capture: content in the manifest before the run; the default, meta, keeps only the hashes.

Selector	Prints
system	The turn's system prompt
tools	The turn's tool schemas
response	The turn's response
message:<i>	Message i (0-based) of the turn's request
a hash, or an 8+ character prefix of one	The body behind that hash directly — no --turn needed

--body writes raw bytes with no trailing newline, so it pipes straight into another command:

mur trace show --body response --turn 2 | tee response.txt | wc -c

---

Step 4 — compare two runs with diff

When you change a manifest, system prompt, or model and want to know if the change helped, use mur trace diff. With no arguments it compares the two most recent sessions automatically:

mur trace diff

Output:

Metric                 Run A            Run B            Delta
────────────────────── ──────────────── ──────────────── ──────────────────────────
turns                  4                2                -2 (B better)
duration               3.2s             1.4s             -1.8s (B better)
input tokens           5,840            2,910            -2930 (B better)
output tokens          612              298              -314 (B better)
input/turn (avg)       1460             1455             -5.0 (B better)
output/turn (avg)      153              149              -4.0 (B better)
tool calls             3                1                -2 (B better)
tool success rate      100.0%           100.0%           =
avg tool latency       28ms             31ms             +3ms (A better)
shell calls            2                1                -1 (B better)
avg shell latency      18ms             22ms             +4ms (A better)
compaction             none             none             —
exit status            ok               ok               —

Lower-is-better metrics (turns, tokens, latency) flag the lower run as better. tool success rate is higher-is-better. Non-comparable fields like exit status show — in the Delta column.

This is the fastest way to confirm that a prompt or model change actually reduced turn count or token usage.

Why did my cache miss — below the metrics table, a Prefix divergence section names the point where the two runs' requests stopped agreeing, which is also the point a provider-side prompt cache stopped matching:

── Prefix divergence ────────────────────────────
system prompt: differs    A d27e9be1c0de…  B aaaaaaaaaaaa…
tool schemas:  identical  143f541e445d…
turn 1:  diverges at message 1  A 4d3fd85ffaa2…  B ffffffffffff…
turn 2:  identical  (2 messages)

It checks the system prompt first, then the tool schemas, then each turn's messages in order — the earliest of these to differ is where the two runs' requests parted ways, and everything after it is a guaranteed cache miss. Both runs need trace.capture meta or content for this section to compare anything.

---

Step 5 — aggregate across sessions

After multiple runs, mur trace report automatically picks up every session in the workdir and aggregates them into a single report:

mur trace report

Output:

Sessions: 5  (./workdir/)

Metric                 Mean           StdDev         Min            Max
────────────────────── ────────────── ────────────── ────────────── ──────────────
turns                  3.2            1.1            2.0            5.0
duration (ms)          2,640          880            1,400          4,100
input tokens           4,720          1,380          2,910          6,850
output tokens          498            142            298            712
tool calls             2.4            0.9            1.0            4.0
tool success (%)       96.0           8.9            80.0           100.0
shell calls            1.8            0.8            1.0            3.0

Exit status:
  ok                       4  (80.0%)
  max_turns_reached        1  (20.0%)

The exit status distribution immediately shows how often the agent hit the turn cap. tool success (%) stddev above 10–15 points usually means the agent is making inconsistent tool choices across runs.

---

Step 6 — read the raw events

trace.jsonl is plain JSONL — each line is one JSON object. You can inspect it directly for detailed debugging:

# All events in order
cat workdir/ses_.../trace.jsonl | jq .

# Only shell events
grep '"event_type":"shell"' workdir/ses_.../trace.jsonl | jq .

# Session summary fields
grep '"event_type":"session_end"' workdir/ses_.../trace.jsonl | jq '{turns: .total_turns, tokens: (.total_input_tokens + .total_output_tokens), status: .exit_status}'

For persistent capsules (queue mode), one trace.jsonl holds one session_start / session_end pair for the launch, with a task_start / task_end block per task inside it. mur trace show displays a per-task breakdown in the Tasks section.

To visualise the same trace data as Grafana spans, see Work with capsule trace spans in Grafana.

---

Summary

Command	Use it for
mur trace show [session]	Quick health check of a single session: turns, tokens, tool errors, latency
mur trace show --body	Print the exact bytes of the system prompt, tool schemas, one message or the response a turn recorded
mur trace steps [session]	Session → task → turn → tool-call breakdown of what the agent did, with per-call latency
mur trace diff [a] [b]	Confirm a change improved (or didn't regress) key metrics; defaults to last two sessions
mur trace diff prefix divergence	Find where two runs' requests stopped matching, when a provider-side prompt cache missed
mur trace report	Spot high variance across runs, find outliers, check exit status distribution
Raw grep + jq	Deep inspection of specific events — which tool calls failed, what commands ran, when compaction fired