Debug Questions Stop Giving Away the Bug

A debug question hands the participant a program with a bug planted in it and asks them to find and fix it. Depending on the assessment, the participant may also have a coding agent that writes and changes code, or Rena, our AI tutor, who gives hints. Whoever writes the question also writes notes for the AI so it understands what the question is testing. On a debug question those notes name the bug and say how to fix it. We have closed several ways that answer could reach the person taking the assessment. We also removed comments that pointed at the bugs from the starter code, the program the participant starts with.

The notes stay on the server

A question's AI notes used to travel to the participant's browser along with the question, and the page sent them back with every AI request. They never appeared on screen, but anyone who looked at the data their browser received could still read the bug and its fix.

Now the notes stay on our server. On other questions, the server adds them to a request just before it calls the AI, and only when the organizer lets the AI see the question. On a debug question it no longer adds them for any AI the participant can use, as the next two sections explain.

The agent no longer knows the answer

On a debug question where the participant had the coding agent, the agent used to get the notes too. Asking it to fix the code handed over the answer, because the agent had been told what was wrong. The question turned into a test of whether someone thought to type "fix it".

The agent no longer gets the notes on any question that asks the participant to fix, clean up or review existing code. It works from the code, the tests and the question text, which is what the participant has. A participant can still ask it to read the code, run the tests and try a fix. The fix it suggests no longer comes from the answer key.

Rena no longer gets the notes either

Rena kept getting the notes on a debug question after the agent stopped, when the organizer let the AI see the question, because they pointed her hints at the real problem. At first they reached her with no label, and when a participant asked what she had been told, she quoted the notes word for word, bug and fix included. We then marked the notes as private and told her never to quote, restate or summarize them. Instructions alone are not a guarantee, so our server also checked every reply for wording copied from the notes.

That was not enough. In one of our own previews of a debug question, asked for her notes, Rena said where both bugs were and what each one did wrong, in her own words. A check for copied wording cannot catch that. So Rena no longer gets the notes on any question that asks the participant to fix, clean up or review existing code, whether or not the organizer lets the AI see the question. Like the agent, she works from the question text, the tests and the participant's code. She has no answer key, so there is none to repeat.

The check on her replies stays as a second guard. If a reply repeats a run of words from the notes, or a piece of code from them, the whole reply is replaced with a short hint. The check runs while the reply appears on screen, holding back its last few words until they have been checked, and once more when the reply is complete.

A refusal says nothing about the guess

If Rena refused only when a guess was right, the refusal itself would give the answer away, since a participant could try guesses one at a time and watch for it. With no notes, she has nothing to check a guess against that the participant cannot also see. A reply the server blocks is replaced with the same fixed text every time. It says she can't give that, then tells the participant to run the tests, pick the first case that fails, and trace their code line by line against the rules in the description until the numbers stop matching.

When Rena repeats words the participant wrote in their own messages, the check lets them through. Blocking only the echoes that happened to match the notes would tell the participant which guess was right.

Starter code that pointed at its bugs

About twenty of the debug questions that ship with AlgoArena had starter code that labeled its own bugs, with comments such as "// BUG: should be a - b" and "// BUG: this line should not be here". A few question texts named the function or the line to change. We removed those comments and sentences from 34 questions and kept the bugs, the tests and the symptoms the text describes. Each changed question was run again to confirm that it still fails with its starter code and passes once the bug is fixed. An automated check in our own code now catches any new or edited debug question whose starter code points at its bugs. Some starters still say how many bugs a file holds, the same count the question text gives, but none points at the line or function that holds one.

Some assessments had saved their own copy of a question before these changes, and that copy still opened with a line saying where one of the bugs was. When the organizer has hints turned off, hints written into a question's text are now removed before the participant sees it.

Why it matters

A debug question measures whether someone can find a problem they were not told about. Each of these leaks let a participant skip that part, but only if they happened to look in the right place. Two people with the same debugging skill could end up with very different scores depending on whether one of them looked at what their browser received or asked Rena a pointed question.