Measuring Project Work, Not Just Answers

Traditional coding assessments are good at one thing: checking whether a final answer passes tests. Modern engineering work asks for more than that.

What changed

We pushed assessments toward project-style work, replayable process, terminal and workspace signals, similarity review, and reporting against benchmarks. The product direction is that if the candidate's process matters, the platform should preserve enough of it to review fairly.

This does not mean every signal deserves equal weight, only that the final score should not be the only artifact. A candidate's planning, iteration, verification, and use of assistance can all help explain what happened.

Why it matters

Two candidates can land on the same final answer for very different reasons. One reasoned through the problem, tested edge cases, and used tools carefully. Another got lucky or followed a brittle path that would collapse in a larger codebase.

Project-style tasks make that difference legible. A larger, multi-step problem has more surface area for judgment, or its absence, to show: where the candidate scoped the work, what they verified before moving on, how they recovered when something broke. A single algorithm answer rarely reveals any of that. A project does, because the path to it is longer and more decisions are visible along the way.

In a project's timeline, the two candidates look nothing alike. One is a run of small moves, each checked before the next, while the other is large blocks of accepted output with a single test run at the end.

Assessments should help teams see that difference, because with AI now normal in software work, the thing worth measuring is judgment about the answer. How to Assess Developers Who Code With AI makes that case in full and walks two such candidates through the same task.

How the process is preserved

The goal is review, not surveillance. A replayable timeline shows how the solution formed, workspace and terminal signals show what the candidate actually ran, and similarity review flags work worth a closer look. A reviewer weighs these together, so a score arrives with the evidence behind it instead of standing alone. A candidate who can explain their path has nothing to fear from it being visible. A result that can't survive that scrutiny was never strong evidence to begin with.

When process signals become noise

Keeping the path has a cost that the cheerful version of this story skips, which is that more signal is also more to misread. A replay timeline can make a careful engineer look erratic if she thinks in long silences and commits in bursts. Similarity review can catch two people who simply studied the same well-known pattern. Terminal history quietly rewards the candidate who narrates every step and penalizes the one who holds most of the work in her head. Left undisciplined, richer evidence just means more ways to arrive at a confident wrong read.

So the work is deciding what a signal is allowed to mean. A flag is a reason to look closer, never a conclusion. The reviewer weighs the trail as a whole and stays honest about its blind spots: a quiet worker is not a weak one, and a talkative one is not automatically strong. The moment a process signal starts behaving like a score, it has stopped being evidence and become a shortcut, which is the exact failure the approach was built to avoid.

Weighting the signals to the task handles many of the false reads, since a burst of reruns means diligence on a debugging task and much less on a greenfield build. AI-Native Assessments Need Evidence, Not Theater covers how role presets move those weights while the report's language stays the same.

A score you can argue with

When the only artifact is a number, you can either trust it or throw it out. Trust it blindly and you hire on a proxy. Throw it out and the exercise was theater. A result carried by its evidence changes that, because a verdict closes the conversation while evidence opens one. The recruiter can see why the score landed where it did. The engineer on the panel can open the path and disagree with it on specifics. The candidate, asked to, can point at the same trail and defend a choice the raw number made look worse than it was, as long as the report is readable by the person it judged. A score you can argue with is worth more than a cleaner one nobody can inspect, because the argument is where the real judgment happens.

Where it points

This work connects to the benchmark direction: clearer standards, better comparison, and less hand-waving around what a score means. The long-term goal is an assessment artifact that a candidate, recruiter, and engineer can all understand. Get that right and the score becomes the first thing a hiring conversation can build on, a shared reference that all three readers can trust even when they come to it for different reasons.