Hidden Tests Stay on Our Servers

Each problem has two kinds of tests. The examples are printed in the statement, and the hidden tests are the larger set that decides whether a submission is accepted. When you submit, the page sends our server your code and the ID of the problem. The server loads the tests, runs your code against them, and sends back the verdict. Practice, battles, friend challenges, interview practice, and coding questions in classroom games and assignments all work this way.

Judging on the server means a result can't be changed from the browser. Your solved problems and practice XP are recorded only after the server has judged the code itself. A battle's result comes from the server's judging of each submission, and a coding answer in a live classroom game scores only what the server recorded for that exact code. It also means a hidden test stays hidden unless it is the first one your code fails, as the next section explains.

What comes back

You get a pass or fail for each test and the number passed. Full details come back for two kinds of test: the statement's examples, which the page already shows, and the first test you failed. For those you see the input, the expected output, your output, and any error message. Every other hidden test reports only whether it passed. Because a program's error output can repeat its input, a crash on any of those tests reads "Runtime error" with no further text, and a test that runs too long reads "Time limit exceeded".

On a practice problem, a failed submission adds a Wrong Answer tab with that first failed test's input, your output, and the expected output, even when the test is a hidden one. The results panel also has a box where you can type a test's number and press Show. It shows any example or the first failed test in full. For any other test it answers that the test is hidden.

Run tries your code on the examples the page shows. Submit is the button that reaches the hidden tests.

The page gets what it shows

The problem page receives a trimmed copy of each problem holding what it displays: the statement and its examples, the time and memory limits, the difficulty, the tags and hints, and how many tests a submission will be graded on. That copy is built from a fixed list of the fields allowed to leave the server, so a field added to problems later stays behind until someone puts it on that list.

Expert Solutions, the reference solutions we publish for each problem, work the same way. Their code is sent only after the server has checked that the account is on Pro. Any other account that is signed in gets an outline with each approach's name, its short summary, and the languages it comes in. The code itself shows as a blurred block marked as part of Pro.

A failed request is not a failed test

Sometimes a request to grade never produces a verdict. It times out, the connection drops, the server answers with an error, or our code runner fails on its own side. Practice used to record that as a Runtime Error in your history, as if your code had crashed. Battles and interview practice marked every test as errored, and a battle also wrote the lower pass count to the match.

Now a request that produced no verdict leaves nothing in your history, your progress, or the match. In place of the results, a notice says grading could not finish, with a Retry button. A real crash in your code still counts as a Runtime Error, because the grader ran your code and reported the crash in its result.

Output in the wrong slot

When one request runs several tests, our code runner occasionally returns one test's output in another test's place. A correct program can then fail a test it actually passed. Before a mismatch counts, the server runs the mismatched tests again and keeps the fresh results. A wrong program prints the same wrong answer the second time, so its verdict does not change. When many of the tests in one request fail at once, the program is taken to be wrong and the server skips the second run.

The runner also sometimes reports one test as timed out when the tests around it finished in milliseconds. That test gets one more run as well, and a program that really is too slow times out again.

Whole numbers match exactly

Decimal answers are compared with a small tolerance, because two correct programs can round differently in the last digit. That tolerance used to apply to whole numbers as well, so 1000000000 was accepted where the right answer was 1000000001. Whole numbers now have to match exactly, while problems that ask for decimals keep their tolerance. A whole number written with leading zeros, such as 007, counts as text and matches only itself.