Keeping AI Features Reliable on Cloud Credits

AlgoArena runs several AI features: candidate assistance, hint coaching, solution analysis, and review copilots. The hard part is not calling a model once. It is keeping every one of those features fast, cheap, and available when a single provider has a bad day.

The problem with one model

A single hard-coded model is a single point of failure. If that provider throttles, errors, or the account balance lapses, every AI surface degrades at the same time. A feature is only as reliable as its least reliable dependency, and we learned that the practical way.

An ordered fallback chain

Instead of one model, each AI surface walks an ordered chain of capable models. The cheapest capable option serves by default. If it is unavailable, the request transparently falls to the next, then the next. A paid, direct API sits at the very end as a last resort that should almost never run.

Two properties make this safe:

Credits first, cash last

Most of the chain runs on cloud platform credits rather than direct, metered spend. That keeps day-to-day inference close to free while a finite credit budget lasts, and reserves real cash for the rare case where every credit-funded option is down at once.

Shipping it safely

The whole behavior sits behind a single master switch. Wiring a route to the chain is a no-op until the switch is on, so we can wire every AI surface first and cut over with one change, then revert just as fast if anything looks wrong.

What this is not

This is not a model-quality claim or a benchmark. It is a reliability and cost posture: keep AI features up, keep default inference cheap, and make a provider outage a non-event instead of a customer-visible failure.