Defining Product-Active Users Across an Arena

Page views are a weak proxy for whether a coding product is working.

AlgoArena has several ways a person can do real work: submit a practice solution, play Puzzle Rush, solve a problem, start an assessment, host a classroom, or answer in a classroom session. A useful product-active metric has to count across those lanes without double-counting the same person.

The definition

For each UTC calendar month, the report counts distinct non-bot account UIDs who did any of:

• submitted code
• submitted a Puzzle Rush answer
• solved a problem
• started an OA session with a real UID
• hosted or played in a classroom game created, started, or ended in that month

Internal and test accounts are excluded before the final count.

The distinct-UID rule is the key. A user who submits 40 times in a month is highly engaged, but they should still count as one product-active user in this metric. Depth belongs in a separate measure. This measure answers the simpler question: how many real people touched a core product loop?

The event families are intentionally broad because AlgoArena has multiple product lanes. A classroom host, a Puzzle Rush player, and an assessment candidate are all doing meaningful work even though their event trails look different.

Why this is better than sign-ins

A sign-in can mean curiosity, debugging, or a forgotten tab. A product-active event means the user touched the learning or evaluation loop.

That does not make the metric perfect. It still misses qualitative depth. But it is much closer to the product promise: people should be practicing, competing, teaching, or being assessed.

The metric also makes product changes easier to interpret. If a marketing page sends more people to sign up but product-active users do not move, the onboarding loop may be weak. If classroom events rise while practice events stay flat, the educator lane may be carrying growth. If assessment starts increase but submissions do not, the candidate flow may be leaking.

Those are product questions, not vanity questions.

What the script outputs

The report produces a JSON sidecar and an HTML visualization. That makes it useful both for quick founder review and for later product notes where a chart needs to match the exact query that produced it.

Keeping the sidecar matters because charts without reproducible inputs decay quickly. A future note should be able to point back to the same aggregation definition and the same excluded-account rules.

What this does not measure

Product-active users are not revenue, retention, learning quality, or hiring accuracy. They are one operational definition of meaningful activity. The metric should be read beside cohort retention, session depth, solve quality, assessment completion, and classroom recurrence.

The value of this definition is that it is stable enough to track monthly and concrete enough to defend.

Method

This note treats the product behavior as the unit of analysis. The useful evidence is the artifact the platform can preserve: the counters, traces, replay rules, validation path, or scoring dimensions that a reviewer can inspect after the session. That keeps the claim grounded in observable work rather than a broad statement about AI, practice, or assessment quality.

Result

The result is a clearer public contract for the feature. A reader should be able to tell what signal is being captured, what product decision it supports, and where the boundary sits. The metric or visual artifact is not meant to be decorative; it is the proof object that makes the note legible.

Limitations

This is not a universal benchmark and it should not be read as one. The numbers and flows here describe AlgoArena's current product surface, current data shape, and current operating constraints. Any future classifier, scoring model, or automated review workflow still needs calibration, false-positive analysis, and human review before it should carry higher-stakes decisions.

Product implication

The product implication is the same across the research archive: make the work easier to inspect without making the experience hostile. Good research packaging should give learners, teachers, and hiring teams a sharper explanation of what happened, while leaving the interface itself calm enough to use repeatedly.

What would make this stronger

The next version should pair this note with more longitudinal evidence: repeated sessions, clearer cohort slices, and failure cases where the current heuristic is not enough. Publishing those limits is part of the credibility. A research surface should not only show the clean path; it should show what would change our mind.