--- title: "Read your agent's trace and compare two runs" description: "Every Murmur agent session produces a trace.jsonl file that captures the complete record of what happened: every inference call, tool invocation, shell command, and compaction event." canonical_url: "https://docs.murmur.nexus/how-to/read-trace-and-diff" last_updated: "2026-08-29T17:34:06.000Z" --- # How to read your agent's trace and compare two runs Every Murmur agent session produces a `trace.jsonl` file that captures the complete record of what happened: every inference call, tool invocation, shell command, and compaction event. This guide shows how to read a single session's trace, spot problems, and compare two runs side by side. The commands in this guide are read-only — they do not modify any file and do not require a running capsule. --- ## Where trace.jsonl lives After `mur run -v`, the session workdir is printed to stdout: ```text murmur: url localhost:52222 session: ses_6801f81dd28b4a9daf434e8324c4793e workdir: /path/to/workdir/ses_6801f81dd28b4a9daf434e8324c4793e manifest: my-agent v0.1.0 driver: murmur-driver-anthropic (claude-sonnet-4-6) ``` The trace lives at `/trace.jsonl`. It is always written — even if the session failed — and cannot be suppressed by the capsule. --- ## Step 1 — print a session summary With no argument, `mur trace show` picks the most recent session automatically: ```bash mur trace show ``` > **Different ways to identify a session** > > `mur trace show` with no argument reads the most recent session: > > ```bash > mur trace show > ``` > > To name another one, pass an ordinal counting back from the newest (`@2`), the last 4 or more > characters of its ID (`3e4b`), the full ID, or a path to its `trace.jsonl`: > > ```bash > mur trace show @2 > mur trace show 3e4b > mur trace show ses_6801f81dd28b4a9daf434e8324c4793e > mur trace show path/to/trace.jsonl > ``` > > Use `--workdir ` if your session directories are not under `./workdir`. Every command > that names a session takes the same addresses — see > [Session addresses](/reference/cli.md#session-addresses). > **Other trace exploration commands** > > `mur trace` has four subcommands for exploring session output: > > **`mur trace show`** — print the full trace for a session to the terminal. > > **`mur trace steps`** — show a turn-by-turn summary of what the agent did in a session. Pass `--verbose` to include a truncated summary of each tool's input. > > **`mur trace diff`** — compare the traces of two sessions side by side, or with no arguments the two most recent. Useful for spotting behavioural regressions between runs. > > **`mur trace report`** — generate a structured summary report from a session's trace. Covers token usage, tool calls, latency, and other session-level metrics. Output: ```text ── Session ────────────────────────────────────── session: ses_6801f81dd28b4a9daf434e8324c4793e capsule: my-agent v0.1.0 model: claude-sonnet-4-6 status: ok duration: 3.2s ── Turns ──────────────────────────────────────── count: 4 (max: 10) ── Tokens ─────────────────────────────────────── input: 5,840 (avg 1460/turn) output: 612 (avg 153/turn) total: 6,452 provider: in 5,720, out 590, cached 4,980, cache write 0 ── Wire ───────────────────────────────────────── turn 0 system bbc5e661e106… tools f9d35d43770d… response afb8c1747105… 1 messages turn 1 system bbc5e661e106… tools f9d35d43770d… response 4ed87cafe960… 3 messages turn 2 system bbc5e661e106… tools f9d35d43770d… response 7c02d4f1ae33… 5 messages turn 3 system bbc5e661e106… tools f9d35d43770d… response 9b4e6a2f0d18… 7 messages bodies: mur trace show --body system --turn 0 ── Tool calls ─────────────────────────────────── count: 3 (3 ok, 0 error) success 100.0% latency: avg 28ms turn 0 bash 22ms ✓ {"command":"cargo build --release"} turn 1 bash 41ms ✓ {"command":"cargo test --workspace"} turn 2 bash 21ms ✓ {"command":"cargo clippy"} ── Shell calls ────────────────────────────────── count: 2 exit codes: 2 ok latency: avg 18ms ── Compaction ─────────────────────────────────── fired: no ``` Each section maps to a class of events in `trace.jsonl`. The turn count and token totals give you the most immediate signal — if a session is taking more turns than expected, the turn count shows it immediately. Under **Tool calls**, each turn row shows the tool that ran, its duration, a `✓`/`✗` status icon, and — when the trace recorded one — the tool's input as compact JSON (truncated to 120 characters). The `provider:` row under **Tokens** appears whenever the driver reported its own counts, and breaks out `cached`/`cache write` tokens so you can see whether the provider's prompt cache actually hit. **Wire** lists each turn's system-prompt, tool-schema and response hashes — not the bytes themselves, which Step 3 covers. Two more sections appear only when they have something to report, both placed so you cannot miss them: - **Hook failures** — printed immediately after **Session**, above every other section. One `✗ ` row per fault, so a broken hook never sits buried below a wall of tool calls. - **Context** — printed when an `on-task-start` hook seeded the task's conversation: the hook's name, whether it committed or was trimmed, the tokens it proposed against its budget, and the ids of the messages it seeded. --- ## Step 2 — spot problems in the output **Too many turns** — if count is close to the maximum (set in `murmur.yaml`, default 10 turns), the agent hit the turn cap. Use `mur trace steps --verbose` to see what it called each turn and what each tool received: ```bash mur trace steps --verbose ``` ```text Session ses_6801f81dd28b4a9daf434e8324c4793e (1 task, 4 turns) task tsk_7a91c2e0… turn 0 tool_call bash tool_call bash 820ms ✓ {"command":"cargo build --release"} turn 1 tool_call bash tool_call bash 1.2s ✓ {"command":"cargo test --workspace"} turn 2 tool_call bash tool_call bash 340ms ✓ {"command":"cargo clippy"} turn 3 end_turn ``` `steps` walks the session → task → turn tree: every turn nests under its task, and every tool call, shell command or skill call nests under the turn that made it. Repeated calls to the same tool with similar inputs, at the same nesting level turn after turn, is the most common sign of a retry loop. **Tool errors** — if the tool calls line shows errors, the model called a tool that returned a failure. Use `mur trace show` to see which tool failed and what input triggered it. **Latency spikes** — `avg tool latency` and `avg shell latency` are per-call averages. `mur trace show` lists every tool call with its individual duration in the Tool calls section, so you can identify the slow call without inspecting the raw file. **Shell failures** — when the agent runs shell commands, `mur trace show` tracks the exit code for each one: ```text ── Shell calls ────────────────────────────────── count: 3 exit codes: 2 ok, 1 failed (exit 1) ``` Non-zero exits are passed back to the model as output, not failures — but if the same exit code repeats across turns, the model is retrying a command that keeps failing. **Compaction that never happened** — a `declined:` row under **Compaction** means a turn crossed the compaction threshold and the context was left alone, so the rest of the session ran over budget: ```text ── Compaction ─────────────────────────────────── fired: no declined: at turn 4 (198,340 tokens) no_hook_replacement ``` `no_hook_replacement` means the capsule declares no compaction hook, or the one it declares returned nothing — declare one to fix it. `unresolved_tool_call` means a hook did return a replacement and the runtime discarded it because its tool calls and tool results no longer paired up; that is a defect in the hook. --- ## Step 3 — print the exact bytes behind a hash `mur trace show` never prints a request or response body itself — only its hash, under **Wire**. To read what a turn actually sent or received, pass `--body` with a selector and the turn it belongs to: ```bash mur trace show --body system --turn 1 ``` Printing a body requires the session to have recorded one — set [`trace.capture: content`](/reference/manifest.md#field-trace) in the manifest before the run; the default, `meta`, keeps only the hashes. | Selector | Prints | |---|---| | `system` | The turn's system prompt | | `tools` | The turn's tool schemas | | `response` | The turn's response | | `message:` | Message `i` (0-based) of the turn's request | | a hash, or an 8+ character prefix of one | The body behind that hash directly — no `--turn` needed | `--body` writes raw bytes with no trailing newline, so it pipes straight into another command: ```bash mur trace show --body response --turn 2 | tee response.txt | wc -c ``` --- ## Step 4 — compare two runs with diff When you change a manifest, system prompt, or model and want to know if the change helped, use `mur trace diff`. With no arguments it compares the two most recent sessions automatically: ```bash mur trace diff ``` Output: ```text Metric Run A Run B Delta ────────────────────── ──────────────── ──────────────── ────────────────────────── turns 4 2 -2 (B better) duration 3.2s 1.4s -1.8s (B better) input tokens 5,840 2,910 -2930 (B better) output tokens 612 298 -314 (B better) input/turn (avg) 1460 1455 -5.0 (B better) output/turn (avg) 153 149 -4.0 (B better) tool calls 3 1 -2 (B better) tool success rate 100.0% 100.0% = avg tool latency 28ms 31ms +3ms (A better) shell calls 2 1 -1 (B better) avg shell latency 18ms 22ms +4ms (A better) compaction none none — exit status ok ok — ``` Lower-is-better metrics (turns, tokens, latency) flag the lower run as better. `tool success rate` is higher-is-better. Non-comparable fields like `exit status` show `—` in the Delta column. This is the fastest way to confirm that a prompt or model change actually reduced turn count or token usage. **Why did my cache miss** — below the metrics table, a **Prefix divergence** section names the point where the two runs' requests stopped agreeing, which is also the point a provider-side prompt cache stopped matching: ```text ── Prefix divergence ──────────────────────────── system prompt: differs A d27e9be1c0de… B aaaaaaaaaaaa… tool schemas: identical 143f541e445d… turn 1: diverges at message 1 A 4d3fd85ffaa2… B ffffffffffff… turn 2: identical (2 messages) ``` It checks the system prompt first, then the tool schemas, then each turn's messages in order — the earliest of these to differ is where the two runs' requests parted ways, and everything after it is a guaranteed cache miss. Both runs need [`trace.capture`](/reference/manifest.md#field-trace) `meta` or `content` for this section to compare anything. --- ## Step 5 — aggregate across sessions After multiple runs, `mur trace report` automatically picks up every session in the workdir and aggregates them into a single report: ```bash mur trace report ``` Output: ```text Sessions: 5 (./workdir/) Metric Mean StdDev Min Max ────────────────────── ────────────── ────────────── ────────────── ────────────── turns 3.2 1.1 2.0 5.0 duration (ms) 2,640 880 1,400 4,100 input tokens 4,720 1,380 2,910 6,850 output tokens 498 142 298 712 tool calls 2.4 0.9 1.0 4.0 tool success (%) 96.0 8.9 80.0 100.0 shell calls 1.8 0.8 1.0 3.0 Exit status: ok 4 (80.0%) max_turns_reached 1 (20.0%) ``` The exit status distribution immediately shows how often the agent hit the turn cap. `tool success (%)` stddev above 10–15 points usually means the agent is making inconsistent tool choices across runs. --- ## Step 6 — read the raw events `trace.jsonl` is plain JSONL — each line is one JSON object. You can inspect it directly for detailed debugging: ```bash # All events in order cat workdir/ses_.../trace.jsonl | jq . # Only shell events grep '"event_type":"shell"' workdir/ses_.../trace.jsonl | jq . # Session summary fields grep '"event_type":"session_end"' workdir/ses_.../trace.jsonl | jq '{turns: .total_turns, tokens: (.total_input_tokens + .total_output_tokens), status: .exit_status}' ``` For persistent capsules (queue mode), one `trace.jsonl` holds one `session_start` / `session_end` pair for the launch, with a `task_start` / `task_end` block per task inside it. `mur trace show` displays a per-task breakdown in the **Tasks** section. To visualise the same trace data as Grafana spans, see [Work with capsule trace spans in Grafana](/how-to/grafana-tempo-spans.md). --- ## Summary | Command | Use it for | |---|---| | `mur trace show [session]` | Quick health check of a single session: turns, tokens, tool errors, latency | | [`mur trace show --body`](/reference/cli.md#mur-trace-show-body) | Print the exact bytes of the system prompt, tool schemas, one message or the response a turn recorded | | `mur trace steps [session]` | Session → task → turn → tool-call breakdown of what the agent did, with per-call latency | | `mur trace diff [a] [b]` | Confirm a change improved (or didn't regress) key metrics; defaults to last two sessions | | [`mur trace diff` prefix divergence](/reference/cli.md#mur-trace-diff-divergence) | Find where two runs' requests stopped matching, when a provider-side prompt cache missed | | `mur trace report` | Spot high variance across runs, find outliers, check exit status distribution | | Raw `grep` + `jq` | Deep inspection of specific events — which tool calls failed, what commands ran, when compaction fired |