Your hiring loop has grown to six interviews. It happened one addition at a time, each for a reasonable reason, and now a candidate meets three engineers, two managers and you.
It feels more rigorous than it did at three. You have added five opinions and roughly one and a half independent assessments, and somebody in that group can now veto anyone without ever saying so.
Why these three models
The decision is who should be on an interview loop and how many people it should contain. The features that fire are a group whose value depends on the independence of its members rather than their number, a set of assessments that must cover what the role actually requires, and a structure that grows by accretion and acquires properties nobody chose.
Three lenses. Aggregation produces a random answer about what makes a panel better than one person. Variety produces an equilibrium answer about what the panel must be able to assess. System traps produce a complex answer about what the loop turns into if nobody designs it. The third explains why hiring gets slower and more conservative as a company grows, without anyone deciding that it should.
1. Independence is the whole mechanism
A group beats an individual under specific conditions: the judgements have to be roughly independent, the errors have to point in different directions, and there need to be enough of them. Meet those and the aggregate is better than almost any member. Fail them and you have one opinion with several signatures.1
Most hiring loops fail the first two conditions comprehensively, and the failures are all procedural rather than personal.
Interviewers read each other’s notes before their own interview, which makes the later assessments conditional on the earlier ones. They discuss the candidate in the corridor between sessions. They are all asked to assess general suitability rather than distinct areas, so they measure the same construct with the same instrument. And the debrief is a discussion in which the most senior or most confident person speaks first.
Each of those is a small convenience and together they collapse six assessments into something close to one. The repairs are equally small. Notes written before any discussion and submitted before any are visible. Distinct assigned areas rather than an overall verdict. A debrief in which everyone’s written view is circulated simultaneously and the most junior person speaks first.
The failure is easy to see once you count it. Six assessors, four of whom read the recruiter’s summary and the first engineer’s notes before their own session. The summary contains the phrase strong communicator. Three of the four use a version of that phrase in their own write-up. You now have four documents agreeing and one source, and the debrief will treat the agreement as evidence. The cheapest test is to open the last debrief you ran, mark every claim that appears in more than one write-up, and find where each one entered. Most of them entered once.
Six people who talked to each other are one opinion wearing six badges.
This is also why the panel size that helps is smaller than people expect. Four genuinely independent assessments of four different things beat eight correlated impressions of the same thing, and cost a candidate half the day.
Independence has a price and it is worth stating so that nobody is surprised by it. Assessors who cannot confer will duplicate ground, and they will lose the follow-up question one interviewer wanted asked because of what another heard. The repair is not to reopen the corridor. It is to move the follow-up into writing after the notes are submitted: any interviewer may name one thing they want probed, and it goes into the next session as a written question rather than as an impression. That keeps the ordering that made the notes independent and still passes the information forward.
2. Can this panel assess what the job needs
The second lens asks a different question from independence: not whether the assessments are separate, but whether they cover the territory.
An assessment process is a regulator, and it can only distinguish situations for which it has a corresponding response. If the role will generate five distinguishable kinds of difficulty and your panel can assess three of them, the process cannot tell you about the other two however many people you add.2
So the design question is not who is available or who wants to be involved. It is what the role must handle, and who in the company can actually judge each of those things. A panel of four engineers assessing a role that is half stakeholder management has excellent coverage of one dimension and none of the other, and it will keep producing confident hires who struggle in the part nobody examined.
The same logic explains a failure that looks like bad luck. A company that keeps hiring people who interview well and struggle in one particular area is not unlucky; it has a panel with no coverage of that area, and it will keep producing the same result until somebody adds the missing assessment rather than another general one.
The exercise is to write the role’s four or five characteristic situations first, then assign each to the person best placed to assess it, and only then count how many interviews that implies. That order produces a smaller loop than the reverse, and a loop where every session has a job.
Coverage is not seniority, and the two get confused because senior people are the ones with the standing to be added. The question is not who is important enough to meet this candidate. It is who has personally handled the situation being assessed. A founder who has never run a collections conversation cannot assess how a finance hire handles one, and being the founder does not change that. Where the person who can judge a situation is junior, that is who should judge it, and the loop should be written around the situations rather than around the hierarchy.
Two repairs are available when the coverage is short and neither is adding another interviewer. Reduce the variety the role has to handle, by narrowing the scope. Or add a genuinely different assessor: a customer, an advisor, a peer from another function, or a paid piece of real work, which assesses things no conversation can.
That last one deserves its own note, because it is the cheapest way out of a coverage gap. Pay the candidate for half a day of the actual work: a real dataset, a real customer email, a real supplier dispute with the names removed. It costs a few hundred thousand shillings and it produces an artefact several people can assess afterwards, separately, at their own desks, which is the one setting where independence is free rather than enforced. It also shows the candidate what the job is, which removes a share of the early departures no interview format prevents.
3. What the loop becomes if nobody designs it
The third lens explains the growth, and it is structural rather than anyone’s fault.
Loops accrete. A bad hire prompts an addition. A team asks to be represented. A new function wants visibility. Each addition is individually reasonable and nobody ever removes anything, so the process only grows.3
What it grows into has two properties nobody chose. The first is an effective veto: with six assessors and an unwritten norm that serious concerns block an offer, any single person can end a candidacy, and the probability that at least one of six finds something rises steeply with the count. The second follows from the first. Under a de facto unanimity rule, the surviving candidates are the ones nobody objects to, which is a different selection from the ones somebody strongly wants.
Both are consequences of an unstated decision rule, which is the thing to fix. Write down who decides, whether any assessment is blocking and which, and what happens when the panel splits. A loop with a named decision-maker and advisory assessments behaves entirely differently from the same people with no rule, and the difference is one paragraph.
The reason this never self-corrects is that nobody ever finds out whether it was working. Almost no company checks its interviewers’ assessments against how the hire actually performed, so a loop can run for years accumulating people without a single piece of evidence about which of them predicts anything.4 You cannot improve a judgement nobody scores.
The other structural fix is a cap. Decide the maximum number of interviews for a role level and hold it, so that adding one requires removing one. Without a cap the process has no mechanism to shrink, and the pressure to add is constant while the pressure to remove is zero.
The cap has a second-order effect you should intend rather than discover. Once adding an interview requires removing one, the argument stops being whether a new perspective would be useful, which is always yes, and becomes which existing session is worth less than the proposed one. That is a comparison people can actually make, and it is the only mechanism that ever removes anything. Put the cap in the role template rather than announcing it in a meeting, because a rule that lives in a document survives the founder being busy.
Expect the removal to be read as a judgement, because it will be. The person whose session you cut hears that their assessment did not count, whatever you say around it. The way through is to cut by situation rather than by person: this loop assesses four things, here is who assesses each, and the sessions not attached to one of the four come out. That is a statement about the design and it can be made in front of everybody. Removing a named individual cannot.
What the three say together
- Write the role’s four or five characteristic situations first, then assign assessors to them. Count the interviews last.
- Independent notes submitted before any discussion, distinct assigned areas, and the most junior person speaks first at debrief.
- State the decision rule: who decides, what is blocking, what happens on a split.
- Cap the loop size for each level, so adding an interview requires removing one.
Where they disagree
The aggregation argument and the coverage argument pull in opposite directions on size.
Aggregation says more independent assessments are better, and the way to get them is more assessors, as long as you protect their independence. The statistical case for a larger panel is real.
The coverage argument says the binding constraint is not the number of readings but whether the panel spans what the role requires, and that six people who can all assess the same dimension add nothing while costing the candidate a week and your team a day.
The reconciliation is that they are counting different things, and coverage should decide the size. Add an assessor only when they cover a situation nobody else can, which is a test each addition can be held to. Under that rule the loop lands at four or five for most roles, the assessments are independent because they are about different things, and the aggregation argument gets what it needs as a by-product rather than through volume.
What none of them contain
None of the three prices the process from the candidate’s side. Six interviews is a week of someone’s life, and the candidates most able to refuse it are the ones with other options, which selects your pool in a direction you did not intend.
None of them handles the interviewer nobody wants to remove. Loops often contain a long-standing member whose assessment adds little and whose removal would be read as a demotion, and that is a relationship problem which no design principle resolves.
None of them handles the first hire into a function nobody here understands. The coverage exercise assumes somebody in the company can assess each situation the role will produce, and for a genuinely new function that is false. The honest response is to borrow an assessor from outside, an advisor or a peer founder, rather than to pretend the coverage exists.
And one property the ensemble will not produce: interviewers get better, and only if someone tells them what happened. Almost no company closes the loop by checking assessments against how the hire actually performed, which means a panel can run for years with no evidence about which of its members can predict anything. That check is cheap and it is the only thing that improves the instrument rather than the process around it.
The one action that survives the ignorance: before the next role, write the four or five situations the job will generate and name who can assess each. If two of your current interviewers map to the same situation and one situation has nobody, you have found both the person to remove and the gap to fill, in twenty minutes.
Who has to move
The founder, because every individual addition to a loop is proposed by someone with a good reason and refused by nobody, which is exactly the structure that only grows. The instinct after a bad hire is to add a stage, which increases correlation and tightens the de facto veto without improving coverage. The cheapest first test is the decision rule written in one paragraph: who decides, what blocks, what happens on a split. Most loops have never had one, and writing it changes the outcome of the next debrief more than any change to who attends.
Sources and notes
- Jeffrey Carpenter and Andrea Robbett, Game Theory and Behavior, MIT Press. The conditions under which aggregated judgements outperform individual ones, and the failure of aggregation when individual estimates become correlated through discussion or sequence, are treated alongside information cascades and social learning. Used in section 1, including the specific consequence that reading earlier notes before conducting your own interview makes the later assessment conditional rather than independent.
- Stafford Beer, Cybernetics and Management, English Universities Press, 1959. The law of requisite variety, attributed to Ashby, is adopted in the chapter on the Black Box as an axiom in the process of controlling complex systems, with the condition stated as sufficient permutative variety to provide a one-to-one transformation from the controlled system into the control box. The two repairs of attenuating incoming variety and amplifying the regulator’s are named as such in Brain of the Firm, Allen Lane, 1972. Used in section 2, applied to whether an assessment process can distinguish the situations a role will actually produce.
- Donella H. Meadows, Thinking in Systems: A Primer, Chelsea Green Publishing. The system traps, and the general finding that a structure acquires behaviour nobody designed when it grows by accretion with no mechanism for removal, are set out in the chapter on system traps and opportunities. Used in section 3 for the emergence of a de facto veto and for the resulting selection toward candidates nobody objects to.
- Michael J. Mauboussin, The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing, Harvard Business Review Press. The treatment of feedback quality, and of why skill fails to develop in domains where outcomes are not linked back to the judgements that produced them, supports the observation in the blind-spot section that interviewers cannot improve without seeing how their assessments compared to what happened.
A note on a number this article does not give. There is no correct number of interviews. The count falls out of the coverage exercise and differs by role, and any benchmark you adopt will be a number that suited somebody else’s job description. Do the situations first and let the count be the output.
Joshua Agonya Pi’Rwot, Founder.