Reuse Beats Regenerate: Teaching Recruiting Copilot to Check the Shelf First

A recruiter asked Recruiting Copilot for something completely reasonable: build an assessment with the analog clock question you already have, and the connect four question you already have. The Copilot searched, found nothing, said so politely, and drafted lookalikes from scratch.

Both questions existed. They had been on the platform for weeks, tested, tuned, and used in real assessments. The agent was not lying about finding nothing. It was searching one shelf in a library with three.

The invisible shelves

Our question inventory grew in layers, and each layer got its own storage. Algorithmic problems live in one catalog. Project questions, the build-a-thing kind that includes the analog clock, live in another. And every company accumulates a third shelf nobody indexed at all: the custom questions its own recruiters authored in past assessments.

The Copilot's search tool covered exactly one of the three, so any request for a question on the other two shelves ended the way the clock and connect four request did. The recruiter sees a competent Copilot doing its best. The platform quietly wastes the work it already did.

Search first, generate second

The fix was to make one search span all three shelves and to teach the Copilot an ordering rule: an existing, tested question beats a freshly generated lookalike every time. A reused question costs nothing to produce, has already survived real candidates, and behaves predictably in grading. A generated one is a first draft with none of that history.

Wider search brought its own risk, that an algorithm query would pull in project questions sharing one word with it. A project question now has to contain every word of the query, so a search for two pointers still returns only algorithm problems. The company shelf is matched on titles the same way, across that company's own recent assessments only. Before telling a recruiter a question does not exist, the Copilot is told to search again with other wording.

The subtle part was labeling, because results from different shelves are included in an assessment through different mechanisms. Each search result now carries instructions for how to actually use it. When the agent finds the analog clock, it also learns how to put it in the assessment it is building.

There is also a proposal-card fix hiding in this story. When the Copilot reuses existing questions, they ride along as references rather than inlined drafts, and the approval card used to count only the drafts. A three-question assessment would present itself as one question, which reads as a bug even when the result is correct. The card now counts both and says how many were reused. The Copilot can also read an assessment it already built, down to the exact question titles, and it drops duplicate titles before saving one.

The general lesson

Copilots inherit the seams of the systems they sit on. Every storage boundary we created for good engineering reasons became a place where the Copilot could honestly find nothing. If you are building on top of accumulated inventory, the Copilot's search tool decides whether you compound your past work or endlessly re-buy it.