Manu's Vault

Calibration System for Performance

3 min readStatus:Seedling

Core Idea

Calibration is the cross-manager process that ensures performance designations mean the same thing across teams. Without it, “Senior Engineer” in Team A drifts away from “Senior Engineer” in Team B, and the entire performance system loses credibility. Larson provides four rules that prevent the most common calibration failures.

Calibration System for Performance

A career ladder and review cycle alone cannot produce fair outcomes. Without calibration, each manager applies the ladder through their own lens — biased by advocacy for their own reports, persuasion skills, and local team context. Calibration corrects this through a shared, structured comparison process.

What Calibration Is

  • A structured meeting where managers review performance assessments across teams
  • Purpose: ensure designations are applied consistently against the ladder, not against each other
  • Participants: managers at the same level (e.g., all engineering managers in a department)
  • Output: designation confirmations, adjustments, and documented rationale

Larson’s Four Rules

  1. Shared quest, not a competition: All participants seek to apply the ladder consistently — not to advocate for their own reports. If managers feel defensive, calibration has become adversarial.

  2. Read, don’t present: Written assessments are shared before the meeting; everyone reads in advance. The meeting is for discussion and comparison — not for managers to pitch their reports. Prevents presentation bias from charismatic or senior managers.

  3. Compare to the ladder, not to peers: The question is always “Does this person’s impact match the level description?” — never “Is this person better or worse than that person?” Prevents punishing engineers on strong teams.

  4. Study the distribution: After calibration, review the population distribution of designations. Warning signs: 80%+ at “Exceeds Expectations” or senior levels far above industry benchmarks. Prevents grade inflation from managers avoiding difficult conversations.

Connection to Designation Momentum

Past ratings create institutional memory. An engineer calibrated as “Exceeds” builds momentum that makes future downward adjustments socially difficult — even when warranted. Applying the rules rigorously from the start matters more than correcting drift later.

Sources

  • Larson, Will (2019). An Elegant Puzzle: Systems of Engineering Management. Stripe Press. ISBN: 978-1-7322651-8-9.

    • Chapter 6.5 — primary source for the four rules of calibration and the calibration meeting structure

  • DeNisi, Angelo S. and Michael K. Murphy (2017). “Performance Appraisal and Performance Management: 100 Years of Progress?” Journal of Applied Psychology, Vol. 102(3), pp. 421-433. DOI: 10.1037/apl0000085.

    • Documents leniency bias, halo effects, and distributional errors; supports criterion-referenced over norm-referenced approaches

  • Colquitt, Jason A. (2001). “On the Dimensionality of Organizational Justice: A Construct Validation of a Measure.” Journal of Applied Psychology, Vol. 86(3), pp. 386-400. DOI: 10.1037/0021-9010.86.3.386

    • Foundational study on procedural fairness (N=776); consistent, bias-suppressed procedures drive perceived fairness

  • Scullen, Steven E., Michael K. Mount, and Maynard Goff (2000). “Understanding the Latent Structure of Job Performance Ratings.” Journal of Applied Psychology, Vol. 85(6), pp. 956-970. DOI: 10.1037/0021-9010.85.6.956

    • Idiosyncratic rater effects account for 62% of variance in ratings — making cross-rater calibration essential

  • Orosz, Gergely (2022). “Performance Reviews for Software Developers – How I Do Them In a (Hopefully) Fair Way.” The Pragmatic Engineer. Available: https://blog.pragmaticengineer.com/performance-reviews-for-software-engineers/

    • Practitioner account of calibration practice at scale; corroborates Larson’s rules from industry experience

Note

This content was drafted with assistance from AI tools for research, organization, and initial content generation. All final content has been reviewed, fact-checked, and edited by the author to ensure accuracy and alignment with the author’s intentions and perspective.

Graph View

Backlinks