Docs

Instrument the agent in this repo with tracehouse so every run is logged
to the tracehouse project "<PROJECT_NAME>" (one trace per run / task).

Setup
- Python: add `tracehouse-sdk` to dependencies, `import tracehouse as cm`.
  JS/TS: add `@tracehouse/sdk`, `import { Run } from "@tracehouse/sdk"`.
- Auth comes from the TRACEHOUSE_API_KEY env var (starts with "ba_").
  Never hardcode the key. Add it to .env.example.

Where to hook in
- Find the agent loop: the place that calls the LLM, parses tool calls,
  executes tools and feeds results back. Wrap ONE full run/task in one trace:
    Python: with cm.Run(project="<PROJECT_NAME>", task_name=<short task title>,
                        model=<model id>) as run:
    JS:     const run = await Run.start({ project, taskName, model })
            … and always call run.finish(...) in finally.
- Concurrent / parallel runs → separate Run objects. Do NOT use the
  module-level cm.init/cm.log_* globals.

Log EVERYTHING, in order, as it happens
- the task / each user or system-injected message → run.log_user(text)
- model reasoning, if the API returns it         → run.log_thinking(text)
- every assistant text block                       → run.log_assistant(text)
- every tool call, with the full arguments dict   → span = run.log_tool_use(name, args)
- every tool result (stringify; keep it whole)     → run.log_tool_result(text, tool=name,
                                                       parent_span_id=span.id)
- a tool that raised: log the error text as the tool result with status="error",
  then continue the agent's normal error handling.
- sub-agents: give them their own Run in the same project.
- JS names: logUser / logThinking / logAssistant / logToolUse({tool, input}) /
  logToolResult({text, tool, parentSpanId}).

Finish
- run.finish(outcome="good" | "bad" | "neutral", metadata={...}) with useful
  metadata: steps, stop reason, final answer length, token usage if available.
  An unhandled exception must still finish the trace with outcome="bad"
  (the Python `with` block does this automatically).

Rules
- Logging must never break the agent: wrap SDK calls so a network error
  is logged as a warning and the run continues.
- Don't change agent behaviour, prompts or outputs — only add logging.
- Keep it in one small module (e.g. tracing.py / tracing.ts) + minimal
  call sites in the loop.
- When done: run the agent once on a tiny task and confirm the trace shows up
  at https://tracehouse.ai/projects/<PROJECT_NAME>.

SDK docs: https://tracehouse.ai/docs

pip install tracehouse-sdk
export TRACEHOUSE_API_KEY="ba_..."   # mint one below (section 2)

import tracehouse as cm

# project = the folder. Same name → same folder; nothing to create first.
# One `with` block = one trace (one agent run) inside that folder.
with cm.Run(project="my-agent", task_name=task) as run:
    run.log_user(task)

    while True:
        reply = llm(messages)                    # your model call, any provider
        if reply.thinking:
            run.log_thinking(reply.thinking)
        if reply.text:
            run.log_assistant(reply.text)
        if not reply.tool_calls:
            break
        for call in reply.tool_calls:
            span = run.log_tool_use(call.name, call.args)
            out = TOOLS[call.name](**call.args)
            run.log_tool_result(str(out), tool=call.name, parent_span_id=span.id)

# Leaving the block closes the trace. An exception marks it outcome="bad"
# with the error in metadata — crashed runs still land in the folder.

import { Run } from "@tracehouse/sdk";

const run = await Run.start({ project: "my-agent", taskName: task });
try {
  await run.logUser(task);
  // … your loop: run.logAssistant(text), run.logToolUse({ tool, input }),
  //   run.logToolResult({ text, tool }) …
  await run.finish({ outcome: "good" });
} catch (e) {
  await run.finish({ outcome: "bad", metadata: { error: String(e) } });
  throw e;
}

pip install tracehouse-sdk

# Required:
export TRACEHOUSE_API_KEY="ba_..."

# Optional — override the default https://tracehouse.ai
export TRACEHOUSE_API_BASE="https://tracehouse.ai"

import tracehouse as cm

cm.init(
    project="my-bot",
    session_id="run-001",          # resumable: same id reuses the trace
    api_key=...,                   # or set TRACEHOUSE_API_KEY env
)
cm.log_user("hello")
cm.log_assistant("hi back")
cm.log_tool_use("Read", {"file_path": "x.py"})
cm.log_tool_result("contents of x.py …")
cm.finish(outcome="good", metadata={"model": "claude-sonnet-4-6"})

import tracehouse as cm

run = cm.init_run(
    project="demo",
    name="qwen-sft-v1",
    config={
        "lr": 1e-4,
        "batch": 32,
        "base_model": "Qwen/Qwen2.5-0.5B",
    },
)

for step in range(1000):
    run.log({"train/loss": loss, "eval/acc": acc}, step=step)

    # Lists / dicts go into a JSON column → rendered as a bar chart.
    run.log({"grad/norm_hist": [0.1, 0.2, 0.4, 0.3]}, step=step)

run.link_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
run.link_model("your-handle/my-finetune")
run.add_artifact("hparams", data={"warmup_ratio": 0.03, "weight_decay": 0.0})
run.finish(status="finished")

# Auto-finish on exit. Exceptions → status="crashed".
with cm.TrainingRun(project="demo", name="exp-42") as run:
    for step, batch in enumerate(loader):
        loss = train_step(batch)
        run.log({"train/loss": loss}, step=step)

import tracehouse as cm

run = cm.init_run(project="demo", name="qwen-sft-v1")

# Log images / videos right next to metrics. cm.Image accepts a file path,
# raw bytes, a PIL image, or a numpy array (Pillow needed only for arrays).
run.log(
    {"loss": loss, "samples": cm.Image("out/epoch3.png", caption="epoch 3")},
    step=3,
)

# Or log media explicitly:
run.log_image("val/grid", "preview.png", caption="val grid", step=10)
run.log_video("rollout", "clip.mp4", step=10)

# cm.Video takes a file path or raw bytes (mp4 / webm / mov).
# Media shows up under the run's Media tab. Limit: 25 MB per item.

# Requires:
#   1. run.link_model(...) earlier so we know the repo
#   2. A write-scope HF token saved in /profile
resp = run.push_model_card(commit_message="Initial card from tracehouse")
print(resp["commit_url"])  # https://huggingface.co/.../commit/<sha>

import logging

# Library doesn't call basicConfig — set it up once in your application.
logging.basicConfig(level=logging.INFO, format="%(name)s %(message)s")

# Verbose HTTP: every → / ← request, byte counts, dropped points.
logging.getLogger("tracehouse").setLevel(logging.DEBUG)

import tracehouse as cm

run = cm.init_run(project="rl", name="ppo-v1", config={"lr": 1e-5})

for step in range(1000):
    # One chat trace per rollout, tied to this run + step.
    with run.rollout(step=step) as t:
        t.log_user(state)
        t.log_assistant(action)
        t.log_tool_result(f"reward={reward}")
    run.log({"reward": reward, "kl": kl}, step=step)   # metrics on the run

run.finish()
# The run page gets a "Rollouts" tab: step 0 → trace, step 1 → trace, …

# Option A — swap the import (new code):
from tracehouse import wandb          # or: import tracehouse.wandb as wandb

# Option B — override existing `import wandb` everywhere, zero edits.
# Put this before the first `import wandb` runs:
import sys, tracehouse.wandb
sys.modules["wandb"] = tracehouse.wandb

# Either way, the usual wandb call sites just work:
run = wandb.init(project="demo", name="qwen-sft", config={"lr": 1e-4})
for step in range(1000):
    wandb.log({"train/loss": loss}, step=step)
    wandb.log({"samples": wandb.Image("out.png")}, step=step)
wandb.config.update({"warmup_ratio": 0.03})
wandb.summary["best_loss"] = best_loss
wandb.finish()

import tracehouse as cm

# Tag a run with three cohort dimensions. Nothing to pre-create:
# the experiment + variant materialize on the first tagged run.
run = cm.init_run(
    project="agent-evals",
    name="run-001",
    subject_id="customer-42",      # the end user this run acted for
    experiment="prompt-rewrite",   # the A/B test name
    variant="treatment",           # this run's arm (vs "control")
    config={"model": "claude-sonnet-4-6"},
)
run.log({"reward": reward}, step=0)
run.finish(status="finished")

# Chat traces take the same dimensions, at init or at finish:
cm.init(project="agent-evals", session_id="s-1",
        subject_id="customer-42", experiment="prompt-rewrite", variant="control")
cm.finish(outcome="good")

import { initRun, init, finish } from "@tracehouse/sdk";

const run = await initRun({
  project: "agent-evals",
  name: "run-001",
  subjectId: "customer-42",        // end user this run acted for
  experiment: "prompt-rewrite",    // A/B test name
  variant: "treatment",            // this run's arm (vs "control")
  config: { model: "claude-sonnet-4-6" },
});
await run.log({ reward }, 0);
await run.finish({ status: "finished" });

// Chat traces accept the same dimensions (camelCase) at init or finish:
await init({ project: "agent-evals", sessionId: "s-1",
  subjectId: "customer-42", experiment: "prompt-rewrite", variant: "control" });
await finish({ outcome: "good" });

# Stats are computed server-side — no LLM, fully deterministic.
# GET the comparison for an experiment by name:
curl -H "Authorization: Bearer $TRACEHOUSE_API_KEY" \
  https://tracehouse.ai/v1/experiments/prompt-rewrite/compare

# -> per-variant cohort aggregates (flag/error/loop rates, cost, latency)
#    + control-relative deltas with Wilson / z / Welch significance.
# The UI renders this at /experiments; per-user rollups live at /subjects.

# One line — detects Claude Code & Codex, installs the binary,
# patches ~/.claude/settings.json, and starts the background daemon.
curl -fsSL https://tracehouse.ai/install.sh | bash

# Equivalent manual path:
tracehouse login --api-key ba_...   # save key + backend URL (once per machine)
tracehouse install                  # add the PreToolUse hook + background service
tracehouse agent                    # start tailing transcripts (`--once` for one pass)

tracehouse              # interactive setup / status TUI (no subcommand)
tracehouse login        # save API key + backend URL locally (wandb-style)
tracehouse install      # patch settings.json hook + install the daemon
tracehouse uninstall    # remove the hook + stop/remove the service
tracehouse agent        # long-running daemon: watch JSONL roots, ship spans
tracehouse agent --once # single sync pass, then exit (good for cron / CI)
tracehouse doctor       # diagnostics: key valid? machine_id? last sync?
tracehouse hook         # internal: PreToolUse handler invoked by Claude Code