Problems That Talk Back

Most coding problems give your program an input and check what it prints. An interactive problem works another way. A hidden program called the judge holds a secret, such as a number or an array, and your program finds it by asking questions one at a time, with the judge answering each one. In some of these problems your program and the judge take turns in a game instead. Our bank has 40 problems like this, and until October 6 none of them could run on the site. The service that runs your code runs one program per test. An interactive problem needs a second program to play the judge, so pressing Run on one of them got an error.

The judge moves inside your program

We made these problems work through Function mode, where you write one function and the site reads the input and passes it in. In an interactive problem's function version, your function also gets a helper object. Each call to one of its methods asks the judge a question or makes a move, and the call returns the judge's answer. The hidden driver, which is the code wrapped around your function, plays the judge inside the same program. It holds the secret, answers each call, counts the calls against the problem's limit and decides the verdict, all in one ordinary run.

Writing every judge again in every language would have multiplied the chances of a mistake, so each problem's judge is written once, in a small language of our own. That judge is compiled into a list of numbers called bytecode. Every language's driver includes the same small interpreter, which reads the bytecode and follows it step by step. A new problem needs no work per language, and a new language needs no work per problem.

Keeping the secret out of reach

The judge's secret sits in the same program as your code, so we had to close the ways your code could simply read it. Before a language is offered for these problems, its driver has to pass a set of checks on the same machines that run your code on the site. The checks exercise every instruction the interpreter knows and every kind of value that passes between your function and the judge. One of them is a test solution that tries to get the hidden test input back by reading its own input again from the start, by opening the files in its folder, and by opening any file another program on the machine holds open. Printing the success message yourself does not pass a test either, because a test passes only when the judge reports success.

The check that tries to read the input back found a real gap. On the machines that run code, Kotlin, Rust, Swift and Elixir get their input in a way that the program's own code can still reach and that no driver can block. The drivers for those four are written and pass every other check, but they stay off for interactive problems until that changes. The other 13 languages are offered: Python, C++, Java, JavaScript, TypeScript, C#, C, Go, Ruby, Dart, Scala, PHP and OCaml.

Checked from the player's side

Each of the 40 problems has our own solutions in Python, C++, Java and JavaScript, and each one passes every test through the same judging the site uses. Each problem also has deliberately wrong solutions, such as one that asks too many questions or one that returns a wrong answer, and the judge has to stop each of them with the right reason. Someone who saw only the statement and the starter code also wrote a solution for every problem, to catch a function version that only makes sense if you already know how the judge was built. One mistake that check found was a problem whose first test steered around the move shown in the statement's example, and we changed the test to replay that example.

A problem can now describe the judge's hidden values in its example tests in one plain sentence. Before, the page listed them by their internal names, like "lead = 5, colors = [1,1,0,0,1]".

On the problem page and in battles

An interactive problem only runs as a function, so the practice page locks it to Function and greys out Full program (where your code reads the input itself), with a tooltip that says why. The four languages without a driver are greyed out in the language picker. If one of them is selected when you open such a problem and you have not typed anything yet, the editor switches to a language that works. Where Function is not offered at all, the error now says the problem only runs in Function mode and that the fault is not in your code.

Battles now have the same Function and Full program picker as practice, in every kind of match, friend challenges included. The choice is saved with your code, so a reload brings your code back in the mode you wrote it in.

We also found two places where a solve in Function mode did not count. Our server checks code again before it records a solve, and those two places now tell it when the code is a function. Before, a battle won with a function was never marked solved, and a classroom assignment solved with a function showed the student Accepted while the teacher saw it as not solved.