Modeling the Cost of Replay Telemetry

Replay sounds expensive until you separate operation count from payload size.

The early match replay design stored periodic code snapshots and minimal event metadata on the match record that already existed. That meant the feature did not need a new write path for every keystroke or a new collection read for every replay. The main cost pressure became storage.

What the model assumed

The 2025 design estimate was:

• 60 to 180 code snapshots per player
• 500 to 2,000 minimal events per player
• roughly 120KB to 1.8MB per two-player match, with about 600KB as the planning average

Those numbers are not a live vendor pricing table. They are a product design model: where will the cost grow if this feature works?

The model separated three costs that are easy to blur:

• writes during the match
• reads when someone opens a replay
• durable storage for retained attempts

That framing changed the design. If replay created a write per keystroke, the feature would be operationally noisy during the match itself. If replay required a large fan-out of document reads, review screens would become slow and expensive. If replay mostly added storage, the team could manage cost through retention, compression, and archival policy.

What it changed

The answer was useful because it shaped the feature:

• do not write every keystroke as a separate document
• do not reconstruct the entire match from raw events if snapshots are enough
• keep replay reads simple for the reviewer
• treat long-term retention as the real lever

The design therefore favored snapshots plus compact event metadata. Snapshots make scrubbing and reconstruction simple. Events preserve enough timing shape to understand how the code changed. The replay viewer can read the match record and render a useful playback without rebuilding the whole session from a document stream.

That is a product tradeoff. It gives up perfect keystroke-level reconstruction in exchange for a more reliable review artifact. For AlgoArena, the question is usually "how did this solution form?" not "can we replay every cursor motion with forensic precision?"

Why storage-first is acceptable

Replay is valuable because it turns process into an inspectable artifact. A learner can see where they got stuck. A reviewer can compare the final answer with the path that produced it. A teammate can debug an integrity issue without guessing from the last code string.

Those benefits require retaining some history. The storage-first model says that retention should be the explicit lever. Keep recent and important replays readily available. Age out or archive low-value data later. Avoid making every match pay a high realtime cost just in case someone opens a replay.

What the figure shows

The figure keeps writes, reads, and payload composition separate. That is the core design decision. The feature should not make the active match path chatty. It should add one durable artifact that can be read when the user or reviewer asks for it.

Product lesson

Replay is an evidence feature. It helps a learner see how their solution formed and helps a reviewer understand process without pretending one final code string tells the whole story.

But replay data should still be boring operationally. The best version is easy to store, easy to retrieve, and easy to age out if the product needs retention limits later.

The lesson still holds as ghost races and assessment review build on top of replay. The more product surfaces depend on process evidence, the more important it is that the underlying telemetry remains simple, explainable, and inexpensive enough to keep.

Method

This note treats the product behavior as the unit of analysis. The useful evidence is the artifact the platform can preserve: the counters, traces, replay rules, validation path, or scoring dimensions that a reviewer can inspect after the session. That keeps the claim grounded in observable work rather than a broad statement about AI, practice, or assessment quality.

Result

The result is a clearer public contract for the feature. A reader should be able to tell what signal is being captured, what product decision it supports, and where the boundary sits. The metric or visual artifact is not meant to be decorative; it is the proof object that makes the note legible.

Limitations

This is not a universal benchmark and it should not be read as one. The numbers and flows here describe AlgoArena's current product surface, current data shape, and current operating constraints. Any future classifier, scoring model, or automated review workflow still needs calibration, false-positive analysis, and human review before it should carry higher-stakes decisions.

Product implication

The product implication is the same across the research archive: make the work easier to inspect without making the experience hostile. Good research packaging should give learners, teachers, and hiring teams a sharper explanation of what happened, while leaving the interface itself calm enough to use repeatedly.

What would make this stronger

The next version should pair this note with more longitudinal evidence: repeated sessions, clearer cohort slices, and failure cases where the current heuristic is not enough. Publishing those limits is part of the credibility. A research surface should not only show the clean path; it should show what would change our mind.