System Design Basics: Building Scalable Applications

System design matters for building applications that can handle millions of users. Here are the fundamental concepts.

Core Principles

1. Scalability

Horizontal scaling means adding more servers.

Vertical scaling means giving each server more CPU, RAM, or disk.

Vertical scaling is simpler, since the code does not change, but one machine has a size limit and takes the whole service down when it fails. Horizontal scaling can keep growing, but only if any server can answer any request. That means servers should keep user state such as login sessions in a shared store instead of their own memory.

2. Load Balancing

Distribute traffic across multiple servers.

• Round robin walks servers in order.
• Least connections prefers the least busy backend.
• IP hashing pins a client IP to a stable backend when you need stickiness.

If you find you need IP hashing, check whether a server is holding state it should share. With sessions in a shared store, round robin works, and a server can crash without logging anyone out.

3. Caching

Store frequently accessed data in fast storage.

import time

cache = {}  # user_id -> (expires_at, user)
TTL_SECONDS = 3600

def get_user(user_id):
    entry = cache.get(user_id)
    if entry and entry[0] > time.time():
        return entry[1]  # hit: no database call
    user = db.get_user(user_id)  # miss: read from the database
    cache[user_id] = (time.time() + TTL_SECONDS, user)
    return user

def update_user(user_id, fields):
    db.update_user(user_id, fields)
    cache.pop(user_id, None)  # the next read fetches the new version

This pattern is called cache-aside. Reads check the cache first and fill it on a miss. The hard part is writes. If update_user did not delete the cached copy, readers would see the old profile for up to an hour. The expiry time is a backstop, since it limits how long any missed delete can serve stale data. A real cache also has a size limit and evicts entries when full, usually the least recently used ones.

Database Design

SQL vs NoSQL

SQL (relational) fits when you need transactions where a group of writes all succeed or all fail (the ACID guarantees), rich joins, and a strict schema you can reason about with the whole team.

NoSQL fits when you need flexible documents, easier horizontal scale-out, or very high write throughput, and you accept different consistency tradeoffs.

Replication

• Primary with replicas. Read from replicas
• Multi-primary. Write to any node

Replicas copy each write a moment after the primary accepts it. A user who saves a profile and reloads may read from a replica that has not caught up and see the old version. A common fix is to read from the primary for a short time after that user writes. Writing to any node is harder. Two nodes can accept conflicting writes to the same row, so you need a rule for which one wins.

Sharding

When writes outgrow one database, split the data across several by a key such as user ID. Choose a key that most queries include, because a query without it has to ask every shard. Watch for hot keys, where one very popular account overloads the shard it lives on.

A cache is usually the cheapest fix for read load. Sharding comes last, because every query then has to know where its data lives, and moving data between shards later is slow and risky.

Key Components

1. CDN (content delivery network) for static assets on servers close to users
2. Message queues for async work (RabbitMQ, Kafka, and similar tools)
3. Microservices when you need to split ownership and deploy independently
4. API gateways as a single front door for clients

A queue lets a request return right away while a worker does the slow part, such as sending an email, later. If the worker crashes, the job stays in the queue and runs again, so the work must be safe to run twice.

A worked estimate

Say a URL shortener gets 100 million new links a month. A month is about 2.6 million seconds, so that is about 40 writes per second. If each link is opened 100 times, reads average around 4,000 per second, with peaks well above that. At 500 bytes per link, five years of links comes to about 3 TB.

Each number points at a decision. Forty writes a second is easy for one database. Thousands of reads a second for the same popular links is what a cache is for. Three terabytes fits on one large machine, so sharding can wait.

Design Process

1. Requirements. Functional and non-functional
2. Capacity estimation for traffic, storage, and bandwidth
3. API design for endpoints and data models
4. Database design for schema and partitioning
5. Component design for the big moving parts
6. Scaling plans for the bottlenecks you expect first

Some interviews also ask you to build a small piece of a system as working code, such as an LRU cache or a rate limiter. The Build a System section of the AlgoArena Lab has practice questions of that kind, checked by running a sequence of commands against what you built.