Repairing a Problem Bank at Catalog Scale

AlgoArena's practice library is large: 10,669 problems drawn from public competitive-programming sources, each shipping with its own stored tests and a pile of stored solutions. The quiet assumption behind a bank like that is that the tests are correct and the accepted solutions pass them. For a corpus this size, assembled over time from many sources, that assumption does not hold. Some tests were wrong. Some stored solutions were wrong. Some problems allowed many valid answers but were judged as if only one string counted. None of it is visible until a real submission hits the bad case, and by then it looks like the platform is broken rather than the data.

This is the story of judging the whole bank against itself and deciding, problem by problem, what to trust.

Method

The starting point was a full re-judge. Every stored solution for every problem was compiled and run against every stored test, at complete coverage across all 10,669 problems. That produces a verdict grid per problem: which solutions pass which tests. A healthy problem is one where the solutions and the tests agree. The interesting problems are the ones where they do not.

Disagreement alone does not tell you who is wrong. A solution can fail because the solution is buggy, or because the test's expected output is corrupt, or because the checker is too strict for a problem that genuinely accepts many answers. So the core of the work was arbitration: for each failing case, weigh the evidence and decide whether the test or the solution was at fault.

The lever that makes arbitration tractable is consensus. A single problem usually carries many independent solutions. If most of them produce the same output on a given input and only the stored expected value disagrees, the expected value is the likely defect, not the solutions. If one solution disagrees with a strong majority, that solution is the likely defect. No single stored artifact is treated as ground truth on its own. The agreement across many of them is.

What arbitration produced

Once a case was arbitrated, the fix followed from the verdict.

Where the expected output was judged wrong, it was regenerated by consensus: run the trusted solutions, take the output they agree on, and write that back as the corrected expected value. Where a problem's local tests were too damaged to trust at all, the fix was to swap in the problem's authoritative upstream test set rather than patch individual cases. Where the solution was wrong, it was retired so it could not sit in the bank as a false example or seed a bad ghost opponent. Thousands of weak or incorrect solutions came out this way.

Some problems allow many correct answers: any valid permutation, any shortest path, any assignment that hits the target. Exact-string judging rejects real solutions on those. For that whole class the repair shipped special-judge checkers, small input-aware programs that read the candidate output together with the problem's input and accept any answer satisfying the stated contract, instead of comparing against one blessed string. More than three thousand checkers went out this way. They run inside a bounded worker pool so a single misbehaving checker cannot hang the judge, and the strongest of them fall back to a deterministic membership test that a model cannot talk its way past.

A subset of problems could not be certified at all: interactive tasks, ones with inconsistent upstream data, ones that timed out even under generous limits. Those were quarantined rather than guessed at. A problem the judge cannot stand behind should be pulled from the catalog, not shipped with a hopeful fix.

Result

The verified plan was applied in one reversible pass. Every document it touched was backed up before it was written, so any change could be restored to its exact prior state. The apply corrected 3,572 problems in production and deployed the special-judge checkers alongside them.

Applying a fix is a claim, not a proof. So the last step was a certification re-judge: run the whole corrected set back through the same judge and confirm each problem now reads healthy. Anything that came back unhealthy after its own fix was a signal that the fix was wrong, and it went back for another look or a rollback rather than staying live on trust.

Limitations

This did not make every problem perfect. It made the bank internally consistent: solutions and tests that agree, checkers that encode the real contract, and a labeled set of problems that were pulled because they could not be certified. Consensus can still be wrong when a whole cluster of stored solutions shares the same bug, so the arbitration favored strong majorities and left genuinely ambiguous cases on exact matching rather than forcing a verdict. A permissive checker is only an improvement while it stays stricter than the problem statement, so checkers shipped only where the contract was clear enough to encode and defend.

The bias throughout was conservative. When the evidence was thin, the repair preferred to quarantine a problem or retire a solution over inventing an answer, and every write stayed reversible in case a later report proves one of those calls wrong.

Product implication

A practice bank is only as trustworthy as its worst silently-broken problem. A learner who writes a correct solution and watches it get rejected does not conclude that one test is corrupt. They conclude the platform is unreliable, and they are right to. The same broken data poisons everything built on top of the bank: ranked battles, ghost opponents, assessment scoring, and any editorial generated from the problem. Repairing the corpus was the unglamorous prerequisite for all of it. The durable posture is the one the certification pass encodes: judge the whole thing against itself, fix only what the evidence supports, keep every change reversible, and re-judge to prove the fix before trusting it.