Manu's Vault

Iterative-Signal-Loop

4 min readStatus:Seedling

The Iterative Signal Loop is the core operational philosophy of Harness-Engineering: every agent failure is treated as a diagnostic signal revealing a missing harness element, not as evidence of model inadequacy. The response to each failure is an engineering act — improve the harness — so the failure cannot recur.

The Loop

Four repeating steps:

  1. Observe failure — an agent makes a mistake or produces an incorrect output
  2. Diagnose the gap — identify what is missing: documentation, a tool, a guardrail, a constraint
  3. Engineer the solution — add or modify the harness element (AGENTS.md update, a new programmatic tool, a linter rule)
  4. Prevent recurrence — the harness now encodes the lesson; the same mistake cannot happen again

Boeckeler (2026) names the loop explicitly: “When the agent struggles, we treat it as a signal: identify what is missing — tools, guardrails, documentation — and feed it back into the repository.” Hashimoto (2026) states the commitment plainly: “Anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.”

Two Types of Harness Improvement

Every loop iteration produces one of two types of output:

  • Documentation update — add instructions, clarifications, or constraints to AGENTS.md / CLAUDE.md so the agent has correct context next time
  • Programmatic tool — build a script, linter, or structured tool that enforces correctness deterministically, removing the burden from the model

The documentation path is fastest; the programmatic path is most durable. Teams often start with documentation and graduate to tools as patterns solidify.

Attribution Shift: Signal, Not Noise

The loop only functions if failures are attributed correctly. Mistele (2026) frames this as the core insight: “The model is probably fine. It’s just a skill issue.” Agent failures that are blamed on the model become dead ends — waiting for a better model. Agent failures attributed to the harness become engineering prompts.

This attribution shift is structurally identical to the quality movement’s reframing: Deming’s PDSA cycle (1986) treats defects as process signals, not worker failures. Rother’s Toyota Kata (2010) extends this into a habitual improvement routine — the improvement kata — where every obstacle is an expected experiment, not a crisis.

Connection to Quality Traditions

The iterative signal loop is a modern application of a 70-year-old quality principle. Yuksel et al. (2025) validate this empirically in a multi-agent context, demonstrating that LLM-driven iterative refinement loops outperform one-shot prompt engineering for complex agentic tasks.

The loop is also how the Rigor-Relocation pattern manifests in practice: rigor moves from up-front model prompting into the feedback mechanism itself.

Sources

  • Boeckeler, Birgitta (2026). “Harness Engineering.” Exploring Generative AI, MartinFowler.com. Published 17 February 2026. Available: https://martinfowler.com/articles/exploring-gen-ai/harness-engineering.html

    • Names the iterative signal loop; establishes the three-category improvement taxonomy; describes the “signal” framing

  • Hashimoto, Mitchell (2026). “My AI Adoption Journey — Step 5: Engineer the Harness.” mitchellh.com. Published 5 February 2026. Available: https://mitchellh.com/writing/my-ai-adoption-journey#step-5-engineer-the-harness

    • Origin of “never make that mistake again” commitment; two-mechanism taxonomy (documentation vs. programmatic tools)

  • Mistele, Kyle (2026). “Skill Issue: Harness Engineering for Coding Agents.” HumanLayer Blog. Published 12 March 2026. Available: https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents

    • Attribution shift framing: “it’s not a model problem, it’s a configuration problem”; validates loop by reframing what failures mean

  • Deming, W. Edwards (1986). Out of the Crisis. MIT Press. ISBN: 978-0262541152. Available: https://mitpress.mit.edu/9780262541152/out-of-the-crisis/

    • PDSA (Plan-Do-Study-Act) cycle as theoretical ancestor; defects as process signals; quality is built in via iterative loops, not inspected in

  • Rother, Mike (2010). Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results. McGraw-Hill. ISBN: 978-0071635233.

    • Improvement kata as habitual four-step loop; every obstacle is an expected experiment; kata framing — the loop must become reflexive practice

  • Yuksel, Kamer Ali, Thiago Castro Ferreira, Mohamed Al-Badrashiny, and Hassan Sawaf (2025). “A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops.” Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025), pp. 52–62. Vienna, Austria. DOI: 10.18653/v1/2025.realm-1.4. Available: https://aclanthology.org/2025.realm-1.4/

    • Peer-reviewed validation: iterative LLM-driven feedback loops outperform one-shot prompt engineering for complex agentic tasks

Note

This note was researched and drafted with AI. How these notes are written →

Graph View

Backlinks