{A}
Back to Research
Judging Jan 9, 2026 5 min read

Validating Problems with Many Correct Outputs

Why exact-output judging breaks on permutation tasks, and how AlgoArena started accepting structurally valid answers instead.

AlgoArena Research

Validating Problems with Many Correct Outputs

2 lines
Accepted shape
any order
Tie groups
optional
Compatibility

Evidence Shape

Some problems need semantic validation: the expected output is a class of answers, not one exact string.

Exact string checksfast but brittle
Input-aware checkshandles valid alternatives
Manual test editsavoided for similar cases
01Parse expected effort
02Parse candidate permutation
03Group equal values
04Accept any valid tie order
Accepted shape: 2 lines (objective value plus permutation) | Tie groups: any order (equal coefficients can swap safely) | Compatibility: optional (input-aware validation preserves old exact checks)

Exact-output judging is simple until a problem statement says "output any valid answer."


One permutation problem exposed the issue clearly. The statement allowed multiple optimal orders when items had equal coefficients, but the sample tests expected different tie orders in different cases. A single correct solution could pass one expected string and fail another.


What changed


The validator learned to use the input when the output shape suggested a permutation-style answer:


  • first line: numeric objective value
  • second line: permutation
  • input: source values that explain which indices are interchangeable

    • If the objective matched and the permutation contained the right indices, items with equal values could appear in any valid order.


    Why this matters


    For learners, a platform that rejects a valid solution feels arbitrary. For a problem library, these failures create hidden debt: the tests look precise, but the precision is fake.


    The fix made the checker closer to the actual problem contract. That is the real standard for judging: not whether the output matches our favorite string, but whether it satisfies the promise made in the statement.


    Boundary


    This was not a universal special judge system. It was a targeted input-aware comparison for a class of permutation tasks. The broader lesson still holds: when a prompt permits many answers, the judge has to understand the answer space.