Sweeping Problems with Many Correct Outputs

Exact-output judging is simple until a problem statement says "output any valid answer."

One permutation problem exposed the issue clearly. The statement allowed multiple optimal orders when items had equal coefficients, but the sample tests expected different tie orders in different cases. A single correct solution could pass one expected string and fail another.

What the sweep measured

The broader June sweep found 399 problems that genuinely accept many valid answers. By June 7, 267 had shipped input-aware checkers. The remaining 132 stayed on exact-match judging because the contract was not safe to encode automatically yet.

Some tasks in that tail were interactive, depended on hidden state, or had inconsistent data. Shipping a checker just to make a chart look complete would make the judge less trustworthy.

The key distinction is "many valid answers" versus "underspecified." A problem can allow multiple correct outputs and still have a precise contract. For example, any valid topological order can be checked against the input graph. But if the statement, examples, and hidden data disagree, a permissive checker can turn real wrong answers into accepted ones.

What changed for shipped checkers

The validator learned to use the input when the output shape suggested a permutation-style answer:

• first line: numeric objective value
• second line: permutation
• input: source values that explain which indices are interchangeable

If the objective matched and the permutation contained the right indices, items with equal values could appear in any valid order.

Other shipped checkers used the same principle: parse the candidate output, compare it to the input contract, and accept any answer that satisfies the stated constraints.

The checkers are input-aware because a candidate's output cannot be judged in isolation. A permutation is only valid relative to the values, graph, ranges, or constraints in the input. The checker has to reconstruct the contract the problem statement promised.

The implementation favors small, readable validators over broad magic. A checker that is easy to inspect is easier to trust when a user reports a surprising accept or reject.

Why this matters

For learners, a platform that rejects a valid solution feels arbitrary. For a problem library, these failures create hidden debt. The tests look precise, but the precision is fake.

The fix made the checker closer to the actual problem contract. The real standard for judging is whether the output satisfies the promise made in the statement.

The issue also matters for AI-native assessments. If the judge rejects valid alternatives, reviewers may blame the candidate or the model for a platform bug. If the judge accepts too broadly, the score can overstate correctness.

How to read the shipped number

Each of the 267 shipped checkers covers a problem where the validator moved closer to the statement. The 132 problems in the exact-match tail are the ones where the safer public behavior was to keep exact matching until the contract can be reviewed.

Boundary and rollback posture

This is not a claim that every "any valid answer" prompt is automatically solved. The published artifact is the shipped sweep: 267 checkers through June 7, an explicit exact-match tail, and a bias toward leaving uncertain contracts alone until they can be verified.

A permissive checker is only an improvement if it is still stricter than the problem statement.

If a checker is found to accept invalid output, it should be rolled back or tightened. If exact matching rejects a valid answer, the problem should move back through contract review. The research artifact is useful because it preserves that operational stance, not because the first sweep is final.

Method

This note treats the product behavior as the unit of analysis. The useful evidence is the artifact the platform can preserve: the counters, traces, replay rules, validation path, or scoring dimensions that a reviewer can inspect after the session. That keeps the claim grounded in observable work rather than a broad statement about AI, practice, or assessment quality.

Result

The result is a clearer public contract for the feature. A reader should be able to tell what signal is being captured, what product decision it supports, and where the boundary sits. The metric or visual artifact is the evidence that makes the note legible.

Limitations

This is not a universal benchmark. The numbers and flows here describe AlgoArena's product, data shape, and operating constraints as they stand today. Any future classifier, scoring model, or automated review workflow still needs calibration, false-positive analysis, and human review before it should carry decisions with higher stakes.

Product implication

The product implication is the same across the research archive: make the work easier to inspect without making the experience hostile. Good research packaging should give learners, teachers, and hiring teams a sharper explanation of what happened, while leaving the interface itself calm enough to use repeatedly.

What would make this stronger

The next version should pair this note with more longitudinal evidence: repeated sessions, clearer cohort slices, and failure cases where the current heuristic is not enough. Publishing those limits is part of the credibility, since a research page should show both the clean path and what would change our mind.