A hiring loop without a rubric doesn’t measure the candidate. It measures whichever interviewer is most confident in the debrief.
That reads like a process complaint, and it isn’t. It’s a measurement problem, and you can watch it happen inside a single meeting.
You see it in a debrief. Two interviewers sat in on the same pairing session, and they arrive with opposite verdicts. One watched the candidate take the keyboard, drive straight to something that worked, and wave off a suggested alternative, and read that as exactly the decisiveness the role is short on. The other watched the same forty minutes, saw someone who didn’t engage with input, and wondered how that lands in a code review six months in.
The uncomfortable part is that both of them are right about what they saw. Neither is careless, and neither is grinding an axe. They came away with genuinely different readings of the same hour, each with real evidence behind it. What they don’t have is a shared idea of what they were looking for, which means the loop has quietly stopped measuring the candidate and started measuring them.
The bar nobody wrote down#
When you don’t define the bar, every interviewer supplies their own from memory. The last hire who worked out. The one who didn’t. The gap they personally had to close in their first six months, now generalized into a requirement. These are reasonable instincts and they’re also unexamined, so two good engineers can hold incompatible ones without either noticing.
It gets worse in the debrief, because debriefs are social. Whoever speaks first sets the frame, the most senior voice carries extra weight, and a confident “I just didn’t get the signal” is remarkably hard to argue with when nobody can say what the signal was supposed to be.
So the rubric’s job isn’t to standardize taste. Taste is most of what you’re paying senior interviewers for. Its job is narrower and more useful: to give words like strong and no signal something to refer to, decided before anyone met the candidate.
That timing is the whole mechanism. A bar you articulate after meeting someone isn’t a bar, it’s your reaction with a rationale attached.
Four stages, and what each one is allowed to decide#
The shape I’ve landed on, after revising it across three companies, is deliberately boring.
A hiring-manager screen comes first. I’m doing that one myself, and I’m not assessing engineering depth in it. I’m checking whether the role as it actually exists matches what this person wants to spend two years doing, because the most expensive hiring mistake isn’t a weak hire, it’s a strong one who leaves in eight months.
Then a technical screen, scoped narrow on purpose. One question it has to answer, and nothing else.
Then a panel of four, each interviewer owning a specific area, assigned in advance. That assignment matters more than it looks. It’s the rubric made structural: coverage becomes a design decision instead of something you hope emerges from four people improvising. It also kills the most common panel failure, which is three interviewers unknowingly probing the same comfortable ground while something important goes unexamined by anyone.
The other thing an assignment buys you is permission to not cover everything. An interviewer who owns one area can go deep in forty minutes. An interviewer who feels responsible for the whole verdict skims five areas and lands on vibes.
The part I changed my mind about#
I used to prefer a take-home exercise, and I still think the case for it is strong.
A take-home lets someone work in their own editor, with their own tooling, at an hour when they’re awake, with nobody watching them type. That’s closer to the actual job than anything else in the loop. It’s also kinder to the people who interview badly and work beautifully, which is a real population and includes some of the best engineers I’ve worked with.
Then I noticed what I was actually getting back. A finished artifact tells you what someone produced. It tells you almost nothing about how they got there, which decisions they weighed and discarded, or what they’d have done with a constraint they didn’t expect. I’d read a clean submission and still not know whether I was looking at someone methodical or someone who’d been down this exact road before.
Collaboration is invisible in a submitted file, and collaboration is most of the job.
So the exercise is live pairing now. The thing I’m watching for isn’t speed or syntax recall. It’s whether someone will say I don’t know yet out loud to a stranger who’s evaluating them, and then say how they’d find out. It’s whether they take a suggestion and build on it, or absorb it silently and route around it. It’s whether they can disagree and still commit inside forty minutes, with a reason attached rather than a preference.
None of that survives the trip into a pull request you read alone on a Thursday.
The switch was about watching people think. What’s happened since is that the authorship signal in a take-home has gotten steadily weaker, which widens the same gap for a different reason. When you can’t be confident who or what produced the artifact, the artifact’s value as evidence drops, and the reasoning you can watch in real time becomes the part worth paying attention to.
Where live pairing is genuinely worse#
It penalizes anxiety. Some strong engineers get measurably worse with someone watching, and pairing charges them for a tax the job doesn’t collect. It rewards people who narrate well, which overlaps with being good but isn’t the same thing. And it’s harder to keep consistent, because now the interviewer is part of the instrument.
Those are real costs, and you pay them down rather than arguing them away. Use the same problem every time so you build a reference class. Tell candidates in advance exactly what the session is, so nobody is surprised into their worst performance. Brief interviewers on how much help to offer, because an interviewer who rescues and an interviewer who watches someone drown are running two different exercises and producing two incomparable scores.
And score the reasoning, not the keystrokes. If your notes could have been written from the diff alone, you’ve just run a slower take-home.
The cheap part#
Designing the rubric is a few hours of argument with people who’ve hired before. Running a loop that measures the wrong thing costs you a year of someone’s output, or a candidate you should have hired going somewhere else with a story about your process.
The rubric is the cheap part. It’s just the part that looks like paperwork, so it’s the one that doesn’t get built.
