Sweeping Problems with Many Correct Outputs

Exact-output judging is simple until a problem statement says "output any valid answer."

One permutation problem exposed the issue clearly. The statement allowed multiple optimal orders when items had equal coefficients, but the sample tests expected different tie orders in different cases. A single correct solution could pass one expected string and fail another.

What the sweep measured

The broader June sweep found 399 genuinely many-valid-answer problems. By June 7, 267 had shipped input-aware checkers. The remaining 132 stayed on exact-match judging because the contract was not safe to encode automatically yet.

That tail is important. Some tasks were interactive, depended on hidden state, or had inconsistent data. Shipping a checker just to make a chart look complete would make the judge less trustworthy.

The key distinction is "many valid answers" versus "underspecified." A problem can allow multiple correct outputs and still have a precise contract. For example, any valid topological order can be checked against the input graph. But if the statement, examples, and hidden data disagree, a permissive checker can turn real wrong answers into accepted ones.

The sweep therefore treated the exact-match tail as a safety outcome, not as unfinished cleanup.

What changed for shipped checkers

The validator learned to use the input when the output shape suggested a permutation-style answer:

• first line: numeric objective value
• second line: permutation
• input: source values that explain which indices are interchangeable

If the objective matched and the permutation contained the right indices, items with equal values could appear in any valid order.

Other shipped checkers used the same principle: parse the candidate output, compare it to the input contract, and accept any answer that satisfies the stated constraints.

This is also why the checkers are input-aware. A candidate's output cannot be judged in isolation. A permutation is only valid relative to the values, graph, ranges, or constraints in the input. The checker has to reconstruct the contract the problem statement promised.

The implementation favors small, readable validators over broad magic. A checker that is easy to inspect is easier to trust when a user reports a surprising accept or reject.

Why this matters

For learners, a platform that rejects a valid solution feels arbitrary. For a problem library, these failures create hidden debt: the tests look precise, but the precision is fake.

The fix made the checker closer to the actual problem contract. That is the real standard for judging: not whether the output matches our favorite string, but whether it satisfies the promise made in the statement.

The issue also matters for AI-native assessments. If the judge rejects valid alternatives, reviewers may blame the candidate or the model for a platform bug. If the judge accepts too broadly, the score can overstate correctness. Both directions hurt trust.

Special judging is not a feature checkbox. It is a commitment to encode the problem's real contract.

How to read the shipped number

The shipped 267 checkers are not "267 problems made easier." They are problems where the validator became closer to the statement. The 132 exact-match tail is not "132 failures." It is the set where the safer public behavior was to keep exact matching until the contract can be reviewed.

This is the same posture used in the product: accept broader outputs only when the rule is clear enough to defend.

Boundary and rollback posture

This is not a claim that every "any valid answer" prompt is automatically solved. The published artifact is the shipped sweep: 267 checkers through June 7, an explicit exact-match tail, and a bias toward leaving uncertain contracts alone until they can be verified.

That is the right product posture for judging. A permissive checker is only an improvement if it is still stricter than the problem statement.

If a checker is found to accept invalid output, it should be rolled back or tightened. If exact matching rejects a valid answer, the problem should move back through contract review. The research artifact is useful because it preserves that operational stance, not because the first sweep is final.

Method

This note treats the product behavior as the unit of analysis. The useful evidence is the artifact the platform can preserve: the counters, traces, replay rules, validation path, or scoring dimensions that a reviewer can inspect after the session. That keeps the claim grounded in observable work rather than a broad statement about AI, practice, or assessment quality.

Result

The result is a clearer public contract for the feature. A reader should be able to tell what signal is being captured, what product decision it supports, and where the boundary sits. The metric or visual artifact is not meant to be decorative; it is the proof object that makes the note legible.

Limitations

This is not a universal benchmark and it should not be read as one. The numbers and flows here describe AlgoArena's current product surface, current data shape, and current operating constraints. Any future classifier, scoring model, or automated review workflow still needs calibration, false-positive analysis, and human review before it should carry higher-stakes decisions.

Product implication

The product implication is the same across the research archive: make the work easier to inspect without making the experience hostile. Good research packaging should give learners, teachers, and hiring teams a sharper explanation of what happened, while leaving the interface itself calm enough to use repeatedly.