{ "module": "0.1 — What Is a Harness?", "course": "Master Course — Harness Engineering", "version": "1.0.0", "duration_minutes": 25, "total_questions": 18, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 4, "application": 7, "analysis": 7 }, "note": "Recall questions verify terminology; application and analysis questions verify the module's load-bearing judgments (thickness spectrum, model/harness attribution, convergent evolution)." }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "Which of the following is NOT one of the three mandatory jobs of a production harness?", "options": [ "Run the model in a loop", "Provide the model with tools", "Enforce safety and permission boundaries", "Fine-tune the model's weights to the task" ], "answer_index": 3, "rationale": "The three jobs are loop, tools, safety. The model inside a harness is typically a general-purpose foundation model — the harness shapes behavior; it does not fine-tune weights. (Teaching doc §2.1.)" }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Approximately what percentage of a production agent system is the AI decision logic (the model), per the MBZUAI study cited in this module?", "options": ["~1.6%", "~15%", "~50%", "~98.4%"], "answer_index": 0, "rationale": "~1.6% is the model; ~98.4% is the harness infrastructure. 98.4% is the harness, not the model — a common reversal error." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "In a typical production session, what makes up the largest share (~67.6%) of what the model sees in context?", "options": ["The system prompt", "Tool outputs", "Conversation history", "Tool definitions"], "answer_index": 1, "rationale": "Tool outputs ~67.6%, tool definitions ~10.7%, history+rest ~18%, system prompt ~3.4%. This is why tool-output formatting dominates over prompt polishing." }, { "id": "Q04", "bloom": "recall", "type": "multiple_choice", "prompt": "Who actually executes a tool call — the model, or the harness?", "options": [ "The model executes the tool directly", "The model emits a tool-call request; the harness intercepts and executes it", "The framework executes it without the harness's knowledge", "Both execute it concurrently for redundancy" ], "answer_index": 1, "rationale": "The model never touches the filesystem, network, or external systems directly. The indirection is a security boundary, not an implementation detail. (Teaching doc §3, Job 2.)" }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You inherit a system with a 5,000-token system prompt, a single LLM call per request, and no tools. The team calls it a 'harness.' By the definition in this module, what is it really?", "options": [ "A thin harness", "A thick harness", "A prompt (not a harness) — no loop, no tools, no safety", "A framework wrapper" ], "answer_index": 2, "rationale": "Loop + tools + safety are all required. A single LLM call with a long prompt has none of the three — it is a prompt (the 'mega-prompt' anti-pattern). Size of the prompt is irrelevant." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "A team reports their agent 'keeps running the same failing tool 10 times in a row and never stops.' Which of the three harness jobs is failing, and what is missing?", "options": [ "Tools — the tool implementation is buggy", "Safety — specifically stop conditions / stuck-loop detection (a loop-control concern)", "Memory — the agent forgot it already tried", "The model — it needs a better system prompt" ], "answer_index": 1, "rationale": "Behavior at iteration boundaries is owned by the loop/safety layer, not the model. The failure is the absence of a stop condition (max retries / stuck-loop detection). Module 7 covers this directly." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You are auditing tool-output size in your harness. Tool outputs average 67% of context. Your team has spent the last two weeks polishing the system prompt. What should you tell them?", "options": [ "Keep polishing the system prompt — it is the highest-leverage work", "Stop. They are optimizing ~3.4% of context while neglecting ~67.6% in tool-output formatting. Redirect to tool-output truncation/formatting.", "Switch to a model with a larger context window", "Add more tools to reduce reliance on any single output" ], "answer_index": 1, "rationale": "The 1.6%/67.6% trap. System prompt is ~3.4%; tool outputs are ~67.6%. The leverage is in tool-output handling, which Modules 2 and 3 cover." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You are applying the 6-step reading method to an unfamiliar harness. Step 2 (find the loop) searches for what?", "options": [ "The main field in package.json", "A while loop or recursion around model.complete() / messages.create()", "JSON schemas with name and input_schema fields", "Token-counting and truncation logic" ], "answer_index": 1, "rationale": "Step 1 = entry point (package.json main). Step 2 = the loop. Step 3 = tool registry (JSON schemas). Step 5 = context manager (token counting)." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "In the n8n loop workflow, execution travels: Trigger → Context → Loop → Model → Route → Tool → Append → back to Loop. What does one full traversal of that ring represent?", "options": [ "One tool call", "One turn of the harness loop (one model call + one tool execution + context update)", "One complete task", "One failed attempt that will be retried" ], "answer_index": 1, "rationale": "The turn boundary is the back-edge from 'Append Tool Result' into 'Loop'. One traversal = one turn = one model call plus the tool execution it requested plus the context update." }, { "id": "Q10", "bloom": "application", "type": "multiple_choice", "prompt": "You are writing the 'Architect's Verdict' (step 6) for Pi. Pi has 4 tools, a <1,000-token system prompt, and trust-the-model permission. Which verdict is correct?", "options": [ "Pi is under-engineered and should add more tools and a permission model", "Pi is thin by deliberate decision — optimized for model-co-evolution as a personal assistant; its minimalism is correct for that use case", "Pi is thin because its developers lacked resources", "Pi is a prototype and should not be studied" ], "answer_index": 1, "rationale": "The thickness spectrum is a design decision, not a quality axis. Pi's minimalism is a deliberate choice the designers defend for the personal-assistant use case. Calling it 'under-engineered' is the module's named failure mode." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "Four teams independently built structurally identical harness architectures. A new hire concludes 'this proves the harness doesn't matter — the model does all the work.' What is wrong with this reasoning?", "options": [ "Nothing — the conclusion is correct", "It inverts the evidence. Convergence on the same harness patterns shows the harness structure is a strong attractor; the 1.6%/98.4% split shows the harness is where the engineering complexity lives.", "The four teams actually coordinated, so the convergence is not independent", "The convergence is coincidence and tells us nothing" ], "answer_index": 1, "rationale": "Convergent evolution of the same three-layer architecture is evidence that the harness structure is a natural attractor — i.e., that the harness matters enormously. The 1.6% model / 98.4% harness split reinforces it." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "A colleague says 'Claude Code is obviously better than Pi — it has 10x more tools and 40x larger system prompt.' Which module concept does this statement violate?", "options": [ "The three-jobs definition", "The thickness spectrum as a design decision, not a quality axis", "Convergent evolution", "The 6-step reading method" ], "answer_index": 1, "rationale": "Treating thickness as a quality score is the named failure mode. Pi and Claude Code are differently optimized for different use cases. Both are correct for their contexts." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "An engineer wraps the OpenAI Assistants API in a thin custom interface and ships it as a 'harness' without reading the SDK internals. When asked how it handles a tool-call failure, they cannot answer. Which anti-pattern is this?", "options": [ "The mega-prompt", "The framework wrapper (black-box framework use)", "The infinite retry", "Tools without schemas" ], "answer_index": 1, "rationale": "Using a framework is fine; treating it as a black box is not. The engineer cannot debug, audit, or secure what they do not understand. (Teaching doc §7, Anti-Pattern 2.)" }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team argues they do not need a sandbox because 'our agent only reads files.' Which of the three harness jobs are they under-investing in, and what is the latent risk?", "options": [ "Loop — the agent may loop forever", "Tools — read_file may need a schema", "Safety — specifically sandboxing and blast-radius containment; 'only reads files' can still mean exfiltration of secrets, traversal, or prompt injection via file contents", "Memory — the agent will forget context" ], "answer_index": 2, "rationale": "Safety is a dimension of every architectural decision. A read tool without a sandbox can exfiltrate credentials, read outside intended scope, or carry prompt-injection payloads. Modules 5, 6, 11 develop this." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "In the 6-step method, why is step 4 (find the safety layer) often the hardest step in practice?", "options": [ "Because safety logic is rarely implemented in real harnesses", "Because safety logic is typically scattered across the loop, the tool executor, and the entry point, rather than centralized in one module", "Because permission models require legal expertise to read", "Because the safety layer is usually written in a different language" ], "answer_index": 1, "rationale": "Teaching doc §6 explicitly notes safety is 'often the least centralized part of the harness.' This scattered property is itself a finding when scoring a harness on the 12-module rubric (Module 6)." }, { "id": "Q16", "bloom": "analysis", "type": "multiple_choice", "prompt": "You are building a harness for a multi-tenant enterprise coding service. You start with Pi's design (4 tools, <1k-token prompt, trust-the-model). What is the strongest critique?", "options": [ "Pi's design is wrong in absolute terms", "Pi's design optimizes for a personal-assistant use case; multi-tenant enterprise demands thick safety (per-tenant isolation, permission flags, auditability) — a thin harness is the wrong position on the thickness spectrum for this use case", "Pi's TypeScript code cannot run in production", "Pi does not have enough tools to be useful" ], "answer_index": 1, "rationale": "The thickness spectrum is a design decision conditioned on use case. Enterprise multi-tenancy pushes the correct position toward thick (Claude Code / NemoClaw end). Pi is correct for its use case, wrong for this one." }, { "id": "Q17", "bloom": "analysis", "type": "multiple_choice", "prompt": "The module argues the model→tool indirection is a security boundary, not an implementation detail. Which of the following best explains WHY it is a security boundary?", "options": [ "Because it makes the code more modular", "Because the harness sits between the model's request and real-world execution, it can enforce permission, sandboxing, and trust-boundary checks at the interception point — the model cannot bypass these", "Because structured output is faster than free text", "Because it lets you swap model providers" ], "answer_index": 1, "rationale": "The interception point is where every permission check, sandbox enforcement, and untrusted-content tag can be applied. Remove the indirection and the model executes directly — no enforcement possible. Modules 2.4, 6, and 11 build on this." }, { "id": "Q18", "bloom": "analysis", "type": "multiple_choice", "prompt": "Two teams report identical benchmark scores. Team A used a thin harness (4 tools, trust-the-model). Team B used a thick harness (40+ tools, 40 permission flags, compaction). A junior engineer says 'the harness doesn't matter, the model is the same.' What is the most important thing this module teaches them they are missing?", "options": [ "Benchmark scores are unreliable", "The thickness decision trades off capability/safety/observability against complexity/failure-modes/attack-surface. Identical scores on one benchmark do not mean the harnesses are equivalent — they are differently optimized, and the right choice depends on the deployment use case, not the benchmark.", "Team B is always safer", "Team A is always cheaper" ], "answer_index": 1, "rationale": "The thickness spectrum's central lesson: every layer adds capability/safety/observability AND code/failure-modes/attack-surface. The senior judgment is matching thickness to use case. A single benchmark cannot adjudicate that." } ] }