What building this has taught us — about hiring once candidates work with
AI, and about what it takes to make an assessment you can actually defend.
No spin, just what we learn, including the parts where we were wrong.
Two kinds of learning live here, and we label them. Building
and Insight notes come from designing and
building the thing — we have plenty of those already, and several of them
cost us features we liked. Findings — what
the judge and the candidates actually do, at what agreement, with what
reliability — need pilots, and the pilots haven't run. We won't publish a
finding we don't have. When we have them, they arrive here with sample
sizes and the things we got wrong.
2026-07-13
Building
We set two AIs on our own product and told them to argue. They found a fairness bug we had shipped.
Not one model reviewing and agreeing — two, from different labs, briefed to disagree with each other and with us. They found that a stage's countdown began the moment the previous stage was submitted. So the clock was running while the candidate sat reading the “nice work, up next” screen. Thinking, stretching, reconnecting: all billed to them, silently. No candidate would ever have reported it, because you cannot notice time you were never given. It is the same lesson the product exists to measure: an AI is worth something when you make it argue and then verify what it says, and worth nothing when you let it agree with you. We found this by being adversarial, not by asking nicely.
ai-review
fairness
2026-07-12
Insight
The hard part of an AI assessment product is not the AI.
The model was the easy bit — it drafts scenarios, it holds a conversation, it scores against a rubric. Nearly all the real work went into the things that guard the comparison: the same challenge for everyone, a clock that only runs while the candidate is actually working, a rubric that cannot move once the first person has sat it, a report that nobody can decide on before it exists. An assessment is a measurement. Most of the engineering is refusing to let the measurement quietly drift.
measurement
product
2026-07-12
Change
We deleted our own flexibility, on purpose.
Scenarios could be drawn at random from a pool, so two people applying for the same role could sit different challenges. It reads like a feature — fresh problems, harder to leak — and it quietly destroys the only thing that makes two scores comparable. Gone. Every stage is now pinned to one scenario, and a campaign with a half-filled plan cannot take candidates at all. Assessment software should be rigid exactly where it matters. We would rather be boring than incomparable.
fairness
product
2026-07-12
Change
You can no longer move the goalposts after the shot.
Compare mode had live rubric sliders: drag a weight, watch the ranking re-sort. It demos beautifully. It is also a machine for motivated reasoning — slide until the candidate you already liked comes out on top, then call it evidence. Weights are now fixed before the first candidate starts, and editing a scenario can no longer re-score someone who has already been assessed. A scoring tool should be a little bit hostile to the person using it. Including us.
rubric
integrity
2026-07-12
Change
We stopped saying “no PII”. It was our best line and we could not defend it.
We never see a name, an email or a CV — there is no field for one. That part is true, and it is rarer than it should be. But we hold a transcript against a persistent token, and that is pseudonymous personal data, not anonymous data. So we rewrote the claim. The precise version turns out to be the stronger one: a reviewer who disproves a single sentence on your trust page goes back and re-reads every other sentence you wrote. Overclaiming is how you cause a long security review.
privacy
honesty
2026-07-12
Change
We deleted our proctoring.
We logged tab-switches and copy/paste and showed reviewers a risk badge. It never surfaced anything a human could not already see in the transcript, it made candidates feel watched, and it was the one thing standing between us and an honest “no browser tracking”. The signal was not worth the surface. The assessment is text, and the record is the work.
privacy
candidates
2026-07-04
Insight
The résumé stopped being a signal.
When every application is written by the same handful of AI tools, the document tells you about the tool, not the person. The thing worth measuring moved from what someone wrote down to how they actually work a problem — which is exactly what an interactive challenge can show.
hiring
signal
2026-06-20
Change
Scenarios are authored per role, not drawn from a bank.
A fixed question bank ages badly and leaks. We moved to scenarios and rubrics authored per campaign from a short brief — so the challenge matches the actual role, the weights reflect what that job needs, and every candidate for it works the same problem under the same rules.
product
rubric