Reuse Beats Regenerate: Teaching the Assessment Agent to Check the Shelf First
Our recruiting agent could draft a question from scratch in seconds, so it did, even when the platform already had a tested version of exactly what the recruiter asked for. Fixing that was a search problem, not a generation problem.
A recruiter asked our assessment agent for something completely reasonable: build an assessment with the analog clock question you already have, and the connect four question you already have. The agent searched, found nothing, said so politely, and drafted lookalikes from scratch.
Both questions existed. They had been on the platform for weeks, tested, tuned, and used in real assessments. The agent was not lying about finding nothing. It was searching one shelf in a library with three.
The invisible shelves
Our question inventory grew in layers, and each layer got its own storage. Algorithmic problems live in one catalog. Project questions, the build-a-thing kind that includes the analog clock, live in another. And every company accumulates a third shelf nobody indexed at all: the custom questions its own recruiters authored in past assessments.
The agent's search tool covered exactly one of the three. So requests that referenced the other two shelves failed in the most convincing way possible: a genuine search, honestly reporting no results, followed by a helpful offer to write a substitute. The recruiter sees a competent agent doing its best. The platform quietly wastes the work it already did.
Search first, generate second
The fix was to make one search span all three shelves, and to teach the agent an ordering rule: an existing, tested question beats a freshly generated lookalike every time. A reused question costs nothing to produce, has already survived real candidates, and behaves predictably in grading. A generated one is a first draft with none of that history.
The subtle part was labeling. Results from different shelves are included in an assessment through different mechanisms, so each search result now carries instructions for how to actually use it. The agent does not just learn that the analog clock exists. It learns how to put it in the assessment it is building.
There is also a proposal-card fix hiding in this story. When the agent reuses existing questions, they ride along as references rather than inlined drafts, and the approval card used to count only the drafts. A three-question assessment would present itself as one question, which reads as a bug even when the result is correct. The card now counts both, and says how many were reused.
The general lesson
Agents inherit the seams of the systems they sit on. Every storage boundary we created for good engineering reasons became a place where the agent could honestly find nothing. If you are building on top of accumulated inventory, the agent's search tool is not a convenience feature. It is the difference between compounding your past work and endlessly re-buying it.
How to read this update
The important question is not only what shipped. It is what new evidence the product can now preserve. For assessments work, a useful release should make the next session easier to understand: what happened, why it mattered, and what someone can do with the result afterward.
That is the bar we use for field notes. A feature is stronger when it turns a hidden process into something reviewable: a timeline, a report, a map, a score explanation, a replay, or a teacher-facing control. The surface can stay simple, but the evidence behind it should get harder to hand-wave.