Kotlin, Scala and Swift answers finish grading
Answers in languages that compile slowly ran out of time while their tests ran, so correct code went to manual review. Each run now compiles once, and long test inputs, large JavaScript programs and a slow code runner no longer fail correct code either.

Our code runner compiles a program again for every test it is given. For most languages that costs little. A Kotlin compile takes about eight seconds there, so an answer with 52 tests took about a minute to run. When other people were running tests at the same time, the request ran past the runner's time limit. The answer was never graded and went to a person to review by hand, even when the code was correct.
One compile for all the tests
We first gave Kotlin, Scala, Swift and Elixir more time and sent their tests in small batches, one batch after another, so no single request reached the limit. That got the answer graded, but it still took a minute.
Now a Kotlin or Swift run sends the program once, with a small driver of ours added to the end of the file and the tests written into it, so the runner compiles it once. When it runs, the driver starts the compiled program again for each test, as a new process with that test's input, the same way the runner would have started it. Every test still runs as a fresh program, so nothing carries over from one test to the next. The student's code is not changed, and line numbers in error messages stay the same.
In our measurements, a run of 52 Kotlin tests now takes about six seconds in one request, where it took about a minute across seven. The same 52 tests in Swift take about a second. Scala's runner already compiles once for all the tests in a request, so Scala went back to sending them together, and 24 tests take about two seconds. A test the driver cannot finish in time goes to the runner the old way.
When we stop waiting for a run, the page now says the run timed out. It used to say the runner was down.
Long inputs
The code runner refuses an input past a certain length. In assessments, a question with one such test failed every test, so correct code scored zero. In practice, full programs in Rust, PHP, OCaml, Dart and Elixir skipped those tests. A test with a long input now goes out with a small launcher that hands the input to the program before the student's code runs, so the program reads it the way it reads any test. These tests now run in assessments and in those five languages.
JavaScript memory
A correct JavaScript program that held about 200,000 small objects crashed with "JavaScript heap out of memory". The runner caps how much memory each process may reserve, and Node had already reserved most of it before the program's first line. Every JavaScript run now starts itself again with a setting that makes Node reserve far less. The same program can now hold about 1.5 million objects. Its output, errors and stack traces read as before.
A slow runner no longer fails a whole run
For eight minutes one morning, about one request in three to the code runner hung for more than 30 seconds, in every language. The same code ran in under a second before and after. Each hung request failed a participant's whole Run Tests or made a submit ask them to try again. A request that times out or gets a server error now goes out again, up to three times in all, while the run still fits its time limit.
Full points only for a full pass
A related fix changes how assessment points round. A question used to round its share of passed tests to whole points, so passing 70 of 71 tests on a question worth 20 points read as 20 out of 20. The report said 100% beside a test count of 70 of 71. A question now gives full points only when every test passed. Older attempts read the same way.
Try it on a real problem
Write code, run it against the tests, and see what passes. No account needed.