Manu's Vault

Iterative-Signal-Loop

4 min readStatus:Seedling

The Iterative Signal Loop is the core operational philosophy of Harness-Engineering: every agent failure is treated as a diagnostic signal revealing a missing harness element, not as evidence of model inadequacy. The response to each failure is an engineering act — improve the harness — so the failure cannot recur.

The Loop

Four repeating steps:

  1. Observe failure — an agent makes a mistake or produces an incorrect output
  2. Diagnose the gap — identify what is missing: documentation, a tool, a guardrail, a constraint
  3. Engineer the solution — add or modify the harness element (AGENTS.md update, a new programmatic tool, a linter rule)
  4. Prevent recurrence — the harness now encodes the lesson; the same mistake cannot happen again

Boeckeler (2026) names the loop explicitly: “When the agent struggles, we treat it as a signal: identify what is missing — tools, guardrails, documentation — and feed it back into the repository.” Hashimoto (2026) states the commitment plainly: “Anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.”

Two Types of Harness Improvement

Every loop iteration produces one of two types of output:

  • Documentation update — add instructions, clarifications, or constraints to AGENTS.md / CLAUDE.md so the agent has correct context next time
  • Programmatic tool — build a script, linter, or structured tool that enforces correctness deterministically, removing the burden from the model

The documentation path is fastest; the programmatic path is most durable. Teams often start with documentation and graduate to tools as patterns solidify.

Attribution Shift: Signal, Not Noise

The loop only functions if failures are attributed correctly. Mistele (2026) frames this as the core insight: “The model is probably fine. It’s just a skill issue.” Agent failures that are blamed on the model become dead ends — waiting for a better model. Agent failures attributed to the harness become engineering prompts.

This attribution shift is structurally identical to the quality movement’s reframing: Deming’s PDSA cycle (1986) treats defects as process signals, not worker failures. Rother’s Toyota Kata (2010) extends this into a habitual improvement routine — the improvement kata — where every obstacle is an expected experiment, not a crisis.

Connection to Quality Traditions

The iterative signal loop is a modern application of a 70-year-old quality principle. Yuksel et al. (2025) validate this empirically in a multi-agent context, demonstrating that LLM-driven iterative refinement loops outperform one-shot prompt engineering for complex agentic tasks.

The loop is also how the Rigor-Relocation pattern manifests in practice: rigor moves from up-front model prompting into the feedback mechanism itself.

Sources

  • Boeckeler, Birgitta (2026). “Harness Engineering.” Exploring Generative AI, MartinFowler.com. Published 17 February 2026. Available: https://martinfowler.com/articles/exploring-gen-ai/harness-engineering.html

    • Names the iterative signal loop; establishes the three-category improvement taxonomy; describes the “signal” framing

  • Hashimoto, Mitchell (2026). “My AI Adoption Journey — Step 5: Engineer the Harness.” mitchellh.com. Published 5 February 2026. Available: https://mitchellh.com/writing/my-ai-adoption-journey#step-5-engineer-the-harness

    • Origin of “never make that mistake again” commitment; two-mechanism taxonomy (documentation vs. programmatic tools)

  • Mistele, Kyle (2026). “Skill Issue: Harness Engineering for Coding Agents.” HumanLayer Blog. Published 12 March 2026. Available: https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents

    • Attribution shift framing: “it’s not a model problem, it’s a configuration problem”; validates loop by reframing what failures mean

  • Deming, W. Edwards (1986). Out of the Crisis. MIT Press. ISBN: 978-0262541152. Available: https://mitpress.mit.edu/9780262541152/out-of-the-crisis/

    • PDSA (Plan-Do-Study-Act) cycle as theoretical ancestor; defects as process signals; quality is built in via iterative loops, not inspected in

  • Rother, Mike (2010). Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results. McGraw-Hill. ISBN: 978-0071635233.

    • Improvement kata as habitual four-step loop; every obstacle is an expected experiment; kata framing — the loop must become reflexive practice

  • Yuksel, Kamer Ali, Thiago Castro Ferreira, Mohamed Al-Badrashiny, and Hassan Sawaf (2025). “A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops.” Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025), pp. 52–62. Vienna, Austria. DOI: 10.18653/v1/2025.realm-1.4. Available: https://aclanthology.org/2025.realm-1.4/

    • Peer-reviewed validation: iterative LLM-driven feedback loops outperform one-shot prompt engineering for complex agentic tasks

Note

This content was drafted with assistance from AI tools for research, organization, and initial content generation. All final content has been reviewed, fact-checked, and edited by the author to ensure accuracy and alignment with the author’s intentions and perspective.

Graph View

Backlinks