Harness end-to-end testing

The unit tests cover the wiring; this guide walks through verifying the real-world integrations that mocked tests can't.

1. Build the harness

1 lines
go install ./cmd/gofastr

gofastr is now on $PATH. Confirm:

1 lines
gofastr harness --help 2>&1 | head -5

2. Real provider tests (ZAI + OpenRouter)

The e2e_real build tag gates Go tests that hit real providers. They skip individually if the relevant env var is missing.

5 lines
ZAI_API_KEY=zai_...                      \
OPENROUTER_API_KEY=sk-or-...             \
OPENROUTER_MODEL=anthropic/claude-3.5-haiku  \
  go test -tags=e2e_real -v -run E2EReal ./framework/experimental/harness \
          -count=1 -timeout 5m

What each test verifies:

TestVerifies
TestE2EReal_ZAI_GLM51ZAI streaming format, usage parsing, full turn lifecycle against GLM-5.1
TestE2EReal_ZAI_ModelsHardcoded catalog still lists GLM-5.1 first
TestE2EReal_OpenRouter_ChatOpenRouter SSE streaming, HTTP-Referer + X-Title headers accepted, usage parsing
TestE2EReal_OpenRouter_Catalog/v1/models parser handles real OpenRouter payload; pricing populated
TestE2EReal_OpenRouter_CacheParseNoCrashCache fields don't crash the stream parser on real provider output

Cost: a typical run is a handful of small completions, single-digit cents across both providers.

3. MCP server via Claude Code

The harness exposes itself as an MCP server over stdio so any MCP-capable client can drive it. To wire it into Claude Code:

Step 1: set the credstore passphrase + provider key

1 lines
export GOFASTR_HARNESS_PASSPHRASE="$(openssl rand -hex 16)"

Then put your provider key into the credstore by running the harness once interactively (it'll auto-create the credstore file under ~/.config/gofastr/harness/creds.enc):

5 lines
# A one-shot interactive run; the credstore is decrypted on first
# access. We'll add a `creds add` subcommand later; for v0.1, put the
# key in via env var that the harness reads on boot:
ZAI_API_KEY=zai_... GOFASTR_HARNESS_PASSPHRASE="$GOFASTR_HARNESS_PASSPHRASE" \
  gofastr harness --prompt "ping"

Step 2: add the MCP server entry to Claude Code's config

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or ~/.config/Claude/claude_desktop_config.json (Linux):

16 lines
{
  "mcpServers": {
    "gofastr-harness": {
      "command": "gofastr",
      "args": [
        "harness",
        "mcp",
        "--profile",
        "/absolute/path/to/framework/experimental/harness/profile/default.toml"
      ],
      "env": {
        "GOFASTR_HARNESS_PASSPHRASE": "your-passphrase-from-step-1"
      }
    }
  }
}

Restart Claude Code. The harness should appear under the MCP server list in the model picker.

Step 3: verify the integration from Claude Code

In a Claude Code conversation:

  1. tools/list shape: ask Claude Code to list MCP tools. You should see gofastr-harness exposing: - harness.create_session - harness.list_sessions - harness.run_agent_with_shell_access, the honestly-named tool - harness.cancel_turn, harness.answer_permission, harness.set_model - harness.enter_plan_mode, harness.exit_plan_mode
  1. Resource listing: ask for resources/list. URIs should appear under the harness/v1:// scheme.
  1. Run-agent smoke test: in Claude Code, ask:

"Use the gofastr-harness MCP server to call run_agent_with_shell_access with prompt 'Reply PONG' on the default session. Wait for the turn."

Expected: Claude Code invokes the tool, the inner harness completes one turn, and the result block contains "PONG" plus a _meta object with cost, turns, toolCalls.

  1. Identity-class enforcement: when the inner agent attempts a tool that triggers a PermissionRequested, the outer Claude Code's harness.answer_permission call should be rejected if it's issued from the same agent that originated the turn. This is hard rule 11 (agents-can't-self-approve) at work.

4. WebSocket transport

8 lines
# Start the harness with the WS listener.
gofastr harness --listen 127.0.0.1:8421 &

# Token issuance requires the interactive confirmation channel (a code
# printed to the harness TTY, see control/auth/issuance.go), so mint
# the token from that interactive session, then connect with `websocat`
# or a small WebSocket client:
websocat -H 'X-Harness-Token: <token>' ws://127.0.0.1:8421/v1/ws?session=sess_...

There is no headless token-file flag yet, so fully non-interactive CI setups must mint interactively once and reuse the token.

5. What this guide does NOT cover

  • Copilot OAuth device flow: requires interactive GitHub auth in a browser. The adapter is built and unit-tested against stubs; real validation needs a separate session.
  • Multi-week chaos testing: concurrency-heavy scenarios that unit tests can't stress.
  • Large-scale cost reconciliation: the local ledger is in place; reconciling against OpenRouter's /credits endpoint at month-end is an operations exercise, not a build verification.

Common mistakes

  • Dropping -tags=e2e_real from the command. The real-provider tests are behind that build tag. Without it they aren't compiled at all, so go test -run E2EReal finds nothing and reports success having tested nothing.
  • Reading a green CI run as provider coverage. Each test skips itself when its key (ZAI_API_KEY, OPENROUTER_API_KEY) is unset. A run with no keys is all skips. Use -v and check for SKIP lines before trusting the result.
  • Omitting -count=1. A cached pass from a previous run can mask a provider-side regression. The whole point of this suite is to hit the real endpoint now.
  • Assuming Copilot is covered. The OAuth device flow needs interactive browser auth; the adapter is unit-tested against stubs only. Real Copilot validation is explicitly out of this guide's scope.