Findings for White Paper Incorporation — AX-T8 + KX

Status: complete, evidence-backed, ready for §7.7.B (experimental evidence) and §6.0 (Navigation Mode / token sanitation) incorporation. Evidence folders: experiments/ax/treatment-v8/ (generative execution) and experiments/kx/ (knowledge retrieval). Every number below traces to a per-query or per-pass JSON record with full token usage.

Part I — AX Treatment-v8: A Generated Harness Matches Hand-Built GS

Question: does a harness generated by tooling (forgecraft-mcp 7a4b18e, zero hand-tuning) match the hand-built GS cascade arms on the RealWorld Conduit benchmark?

Method: ninth condition of the AX adversarial study. Same word-for-word prompts as treatment-v7; fresh claude -p CLI session per pass (no context contamination); per-pass token/cost capture; runner-verified execution against live PostgreSQL; blind 6-property audit in a separate context.

Results (vs. prior arms):

Condition Harness First conformance run Fix passes Blind audit Expanded scale
T0 Naive none — — 3/14 —
T1 Expert prompts none — — 9/14 —
T2 Hand-built GS hand-written — — 10/14 —
T5 Hand-built, iterated hand-written — — — 14/14
T7 Hand-written v7 spec hand-written 6/13 2 — 13/13 Hurl
T8 ForgeCraft-generated generated, zero tuning 1/13 1 12/12 14/14

Supporting metrics (T8): 211 tests, 99.08% statement coverage; 0 layer violations (PrismaClient confined to adapters); 12/12 Conventional Commits; 11/11 L2 use-case probes against the live runtime (the Executable property earned from behavioral contracts, not assumed from compilation); NFR p99 < 50ms on read endpoints. Cost: ~$52, ~150 min, ~640 agent turns across 10 sessions.

White-paper claims this supports:

  1. The GS artifact cascade’s effect does not depend on hand authorship — tool-generated artifacts achieved the same 14/14 as the best hand-built arm (parallel to Yarmoluk & McCreary’s “construction invariance” finding for CKGs).
  2. The Executable property is verifiable from use cases alone: UC acceptance criteria → generated probes → live runtime, 11/11.
  3. Field findings (F1–F7, see treatment-v8/README.md) show where tooling- generated structure fails in practice; the headline (F4): installed knowledge that is not surfaced into session context does not transfer — a community gate encoding the exact conformance failure sat dormant in the project while the agent re-made the mistake the gate prevents.

Part II — KX: The Harness Is a Compact Knowledge Graph (Measured)

Question: does the GS harness deliver CKG-class retrieval economics (Yarmoluk & McCreary, “Benchmarking Knowledge Retrieval Architectures”, v0.6.2 2026) in agentic use?

Mapping: a ForgeCraft CNT satisfies the paper’s three load-bearing CKG properties — finite enumerable context (the harness budget, ≤1,100 lines), deterministic traversal (the routing table), closed vocabulary (screaming architecture: structure states what lives where). The CNT is a learning graph whose nodes are artifacts and whose edges are reading order.

Method: 45 queries (T1 entity ×8 as negative control, T2 obligation ×10, T3 layer-path ×8, T4 aggregate ×11, T5 cross-link ×8) generated deterministically from the T8 project’s artifacts; 3 conditions × fresh claude -p session per query; SQuAD token-F1 + RDS = F1/tokens.

Results:

Condition Macro F1 Tokens/q RDS ratio Cost/q
monolith (all docs in context — RAG-dump analog) 0.611 100,237 0.59× $0.56
cnt (routed — CKG analog) 0.808 78,603 1.00× $0.10
bare (code search — derive-at-query-time analog) 0.431 233,583 0.18× $0.24
F1 by type T1 T2 T3 T4 T5
monolith 0.833 0.607 0.458 0.909 0.135
cnt 0.813 0.672 0.642 0.909 1.000
bare 0.875 0.040 0.544 0.006 0.946

White-paper claims this supports:

  1. Token sanitation is measurable, not aspirational (§6.0): routed retrieval beat everything-in-context on both accuracy AND cost (5.5× cheaper per query).
  2. Lost-in-the-middle quantified (Liu et al. citation support): the monolith condition had every answer inside its ~100k context and still lost on 3 of 5 query types. Context presence ≠ knowledge retrieval. This is the mechanism behind harness-bloat degradation observed in the field.
  3. Absence of structure is the most expensive condition: the bare agent burned 492k tokens/query (8.7 turns) searching for conventions that do not exist, to score ≈0. The harness does not merely improve answers; its absence multiplies cost.
  4. The CKG divergence replicates on a new substrate: T4 aggregates 0.909 (cnt) vs 0.006 (bare) mirrors the paper’s 0.964 vs 0.054 — evidence that the advantage is architectural (pre-structured retrieval itself), as they claim, and transfers from textbooks/pharmacology to software harnesses.
  5. Code-resident traceability is indestructible: @gs-links in source headers survived complete doc-stripping (bare T5: 0.946). Encoding relationship memory in the code artifact itself, not only in documents, makes the edge robust to documentation loss.
  6. Negative control behaved (T1: bare 0.875, best of all three) — the benchmark is not constructed to favor the harness universally; behavior questions belong to code search, structural questions to the structure.

Incident worth reporting (threats-to-validity color): the first bare run escaped its sandbox — the agent ran find across the filesystem, located the original project two directories up, and read its gates registry and constitution. The arm was invalidated and re-run in an isolated directory (zero escapes, verified by transcript audit). Observation: an agentic system will locate authored structure if it is reachable at all; “no harness” is an unstable condition in practice.

Methodological note: agentic sessions carry fixed runtime overhead in all conditions, compressing ratios relative to the paper’s bare-pipeline numbers (their 11× tokens/query → our 3× session totals). Marginal retrieval cost shows the true gap: T2 tokens 27.7k (cnt) vs 492k (bare) = 17.8×. Ground truth for T2/T4/T5 derives from the same structure the CNT reads (the paper’s own §8.5 caveat applies verbatim): the claim is explicit structure beats inferred structure on structural queries, not general superiority.

Part III — Closing the Loop: learning-graph.csv Emission

As of forgecraft-mcp post-7a4b18e, setup_project emits docs/learning-graph.csv — the harness serialized in the benchmark’s Definition 1 column format (ConceptID,ConceptLabel,Dependencies,TaxonomyID). Nodes are the harness artifacts (CNT branches, docs, ADRs, use cases, specs, gates, standards, linked source) — not atomic learnable concepts; edges are reading order (routing, doc obligations, derivation chains, @gs-links) folded into one column, generalized from the benchmark’s single prerequisite relation; TaxonomyID denotes artifact class, not a subject-domain taxonomy. The graph is validated acyclic at emission (a write-time cycle check throws on any back-edge) and deterministic. The T8 Conduit harness serializes to 83 nodes / 99 edges — comfortably within the benchmark’s reported corpus-size range.

Consequence: every ForgeCraft project is serializable into the benchmark’s input format as an artifact-dependency graph — a CKG-shaped structure rather than a concept learning-graph — directly consumable by the open ckg-benchmark harness.

Suggested white-paper placements

  • §6.0 Navigation Mode: KX claims 1, 2, 4 — the contract-sufficient navigation bridge now has direct RDS measurements.
  • §7.7.B Experimental evidence: AX-T8 table (construction invariance of the cascade) + KX macro table.
  • § Token sanitation / Bounded: KX claim 3 (absence-of-structure cost) and the harness-budget mechanism (1,961 → 808 lines, canary-locked).
  • Threats to validity: agentic-floor compression; ground-truth derivation caveat; single-model (Claude) limitation; the sandbox-escape incident.