Manu's Vault

Seeking-Wrong-Goal

4 min readStatus:Seedling

A system pursues a goal efficiently — and achieves exactly the wrong thing. Seeking the wrong goal is the system trap where the problem isn’t the feedback structure, the actors, or the resources: it’s the goal itself.

The Trap Structure

Every goal in a real system is a proxy. Wellbeing gets measured as GDP. Education quality gets measured as test scores. Programmer productivity gets measured as lines of code. The proxy is never identical to the underlying purpose. When the gap between proxy and purpose is small, the system works. When the gap grows — or was never small — the system optimises faithfully toward the wrong target.

What makes this trap distinctive: the system is working correctly. Balancing feedback loops close the gap between state and goal. The result converges. The mechanism is fine. The problem is entirely in the goal specification.

Why Capability Amplifies Damage

A weak system pursuing the wrong goal drifts slowly. A powerful, efficient system pursuing the wrong goal accelerates toward the wrong outcome and arrives there faster with more resources spent. Greater capability increases both the speed and the magnitude of the damage. This is why well-funded, well-managed institutions can produce worse outcomes than under-resourced ones: they are more efficiently optimised to the wrong target.

Distinct from Rule-Beating

Rule-Beating involves intentional gaming — actors know the rule is a proxy and exploit the gap. Seeking the wrong goal involves sincere misalignment — actors genuinely believe they are pursuing the right thing. The system serves its stated goal in good faith. The error is architectural, not motivational.

Examples Across Domains

  • Economic policy: GDP growth as a proxy for societal wellbeing drives decisions that increase output while eroding health, equality, and environmental stability
  • Healthcare: Hospital readmission rates as a quality proxy can incentivise discharging patients later (not better), or avoiding complex patients who might return
  • Organisations: Quarterly earnings as a proxy for firm health incentivises short-term decisions that damage long-term competitiveness

Escape Routes

  • Use multiple measures: A single proxy is always divergent from purpose; triangulate across several imperfect measures that capture different aspects of the true goal
  • Build feedback from real outcomes: Connect decision-makers to observable consequences of their decisions, not just to the proxy metric
  • Examine goals explicitly: Treat goal-setting as a design activity — ask “if we achieve this goal perfectly, have we served the purpose?”
  • Revisit goals over time: Purposes evolve; proxies drift; goals need active review, not just periodic updating

Sources

  • Meadows, Donella H. (2008). Thinking in Systems: A Primer. Chelsea Green Publishing. ISBN: 978-1-60358-055-7.

    • Chapter 5, pp. 145-151: seeking the wrong goal as a system trap

  • Muller, Jerry Z. (2018). The Tyranny of Metrics. Princeton University Press. ISBN: 978-0-691-17472-5.

    • Comprehensive cross-domain analysis of how metric fixation drives goal displacement in education, medicine, policing, and business

  • Ariely, Dan (2010). “You Are What You Measure.” Harvard Business Review, June 2010.

    • Practitioner account of how choosing the wrong metrics reshapes organisational behaviour in unintended directions

  • Stiglitz, Joseph E., Amartya Sen, and Jean-Paul Fitoussi (2010). Mismeasuring Our Lives: Why GDP Doesn’t Add Up. The New Press. ISBN: 978-1-59558-519-6.

    • Commission report on why GDP systematically fails as a proxy for human welfare and what multidimensional alternatives look like

  • Ordóñez, Lisa D., Maurice E. Schweitzer, Adam D. Galinsky, and Max H. Bazerman (2009). “Goals Gone Wild: The Systematic Side Effects of Over-Prescribing Goal Setting.” Academy of Management Perspectives, Vol. 23, No. 1, pp. 6-16. DOI: 10.5465/AMP.2009.37007999.

    • Empirical review of how narrowly-defined performance goals produce unethical behaviour, risk-taking, and strategic short-termism

Note

This content was drafted with assistance from AI tools for research, organization, and initial content generation. All final content has been reviewed, fact-checked, and edited by the author to ensure accuracy and alignment with the author’s intentions and perspective.

Graph View

Backlinks