Free for classroom and personal use under CC BY-NC 4.0.
Vibecoding1. Introduction to Vibecoding and the AI Ecosystem
1. Introduction to Vibecoding and the AI Ecosystem
What vibecoding is, where it came from, and the tools people use to do it.
~13 min read
VibecodingVibecodingBuilding software by describing intent in natural language and steering an AI toward a working result, instead of hand-typing every line. means building software by describing what you want in natural language and letting an AI produce the code. In practice, the word covers a wide range. Some people fire off one prompt and ship whatever comes back. Others specify carefully and check every result. This book teaches the second kind, and this chapter maps the tools and the vocabulary you'll need for it.
By the end of this chapter, you will be able to
Define vibecoding and explain its four-step loop: intent, generation, verification, refinement
Summarize the major milestones in AI-assisted programming, from autocomplete to agentic IDEs
Tell chat assistants, AI-native IDEs, and app builders apart, and know when to use each
Recognize a ChatGPT wrapper and determine whether it adds real value beyond the base model
Choose a model that fits the job, from fast and cheap to slow and powerful, and know when to check whether better options have shipped
Compare pros and cons of frontier versus fast model tiers and choose appropriately for a task
Explain why human verification becomes more important as AI tools gain autonomy
1.1What's vibecoding?
VibecodingVibecodingBuilding software by describing intent in natural language and steering an AI toward a working result, instead of hand-typing every line. is a workflow where you say what you want in plain language, direct an AI model to write the code, and check, correct, and iterate until it actually does what you intended. The term was popularized in February 2025, when Andrej Karpathy, a founding member of OpenAI and former head of AI at Tesla, described 'fully giving in to the vibes' and forgetting that the code even exists [1]. It spread because it named something a lot of people had quietly started doing: building by conversation.
The reason vibecoding spread is speed. Even an engineer who could write the same code by hand can't do so as fast as a model, so a draft that would have taken an hour arrives in minutes flat. The deeper win comes next, because while the model works, you're free to write the next prompt or steer a second agentAgentAn AI system that can take actions, see what happened, and decide what to do next, rather than only producing text. on another task. Your output is no longer limited by how fast you produce code, but by how fast you can decide what to build and confirm it was built right.
The careless end of that range is what gives vibecoding its reputation, and the difference between the two ends is the difference between a demo and a product. This book is aimed at the product end, which means you stay responsible for the result while the AI produces it quickly, tirelessly, and sometimes wrongly. A demo only has to work once, in front of people rooting for it. A product has to work at 3 a.m. for a stranger who isn't.
You'll sometimes hear 'vibecoder' used as an insult, shorthand for someone who can't really program and just pastes whatever the model says. Reject that framing, because every generation of tooling has drawn this line and then witnessed it move: assembly programmers doubted compilers, and C programmers doubted garbage collection. The question is whether the work holds up and whether the person who shipped it can stand behind it. A vibecoder who specifies clearly and verifies carefully is doing real engineering, which is the standard this book holds you to.
None of this means programming knowledge stops mattering, only that it shows up differently. You still need to recognize what 'correct' looks like and why an off-by-one breaks a loop, because those are the kinds of judgments you'll constantly be making about the model's output. What changes is where your attention goes, moving away from recalling syntax and toward specifying and verifying.
You plan and architect the build, you direct and steer the model as it generates the code, you check that the result holds to your intent, and after that check you draft your next request for code before the cycle repeats.
The four-step vibecoding loop, in which you plan and architect the build, direct and steer generation, verify the result against your intent, and after verification draft the next request for code before the cycle repeats.
1.2A short history of AI-assisted programming
AI assistance in code editors didn't arrive all at once, and seeing the steps makes today's tools less mysterious, starting with the autocomplete and IntelliSense that shipped in Visual Studio back in 1997. These were rule-based and statistical suggestions scoped to a single identifier or line, driven by the symbols already defined in your project. Useful, but they only ever finished a thought you had already started typing.
The next wave was search, meaning Google and Stack Overflow, where the human still did most of the reasoning manually, while the machine only did retrieval, until the real change came in 2021 with GitHub Copilot. It was the first widely used tool to put a large language modelLarge Language Model (LLM)A neural network trained on vast amounts of text to predict likely continuations, which is what lets it generate code, prose, and explanations. (LLM), a network trained on vast amounts of public code, directly in the editor, suggesting whole multi-line blocks from only a comment or a function name.
From there, chat-based assistants (ChatGPT in late 2022, Claude in early 2023) could hold a conversation about an entire problem, explain their reasoning, and adapt across many turns. Then agentic IDEs (Cursor, Claude Code, Devin Desktop) narrowed the gap further by letting the model read, edit, and run code across many files on its own. Given permission, these tools reach past the editor and onto your machine, which turned 'suggest some text' into 'do the task.'
The pattern is worth naming because it predicts what comes next. Each wave automated more of the how and left the human more of the what and the question of whether the result is right, and vibecodingVibecodingBuilding software by describing intent in natural language and steering an AI toward a working result, instead of hand-typing every line. is the current step in that progression. Specifying and verifying are the skills that every wave so far has left in human hands.
1.3Chat assistants
ChatGPT, Claude, and Gemini are general-purpose chat assistants. You should keep in mind that the product is separate from the model within it. For example, ChatGPT is the product, while the LLMLarge Language Model (LLM)A neural network trained on vast amounts of text to predict likely continuations, which is what lets it generate code, prose, and explanations. is only one part of a package that also includes the interface, the saved instructions, and any tools it can access. These assistants answer questions, explain code, brainstorm designs, and write snippets. Most can now run code in a sandbox of their own, and some can reach your files if you connect them. What stays true is that, by default, they work on whatever you paste in, or on what they have produced themselves in that conversation, not on your project on your machine. That boundary is useful because it makes a chat assistant a low-stakes place to think before you commit to anything.
So the best use of a chat assistant is often before you write any code. Pressure-test an approach, compare two code libraries, or ask it to poke holes in your plan. It's a better thinking partner for the what and the why than a code generator for the how, which is what the IDE-level tools do better anyway.
Each of these products is really a family of models at different tiers of capability and cost [3][5][6], from a fast, cheap one for easy requests up to a larger, slower one for hard reasoning. Using the most expensive model to change a button color could waste money, while using the cheapest to design a database schema could waste your afternoon fixing its mistakes.
One limit is worth stating up front. A chat assistant only knows what's in the conversation plus what it learned during training, which stopped at a fixed date. Ask it about a library version released last month, and it will often answer confidently about the version it remembers. Most assistants can now search the web, which helps a great deal with anything recent, though it introduces a failure of its own, because a model can't reliably tell a good source from a bad one and will summarize either with the same confidence. This is a first taste of the hallucination problem that Chapter 9 makes central, and of the context questions that Section 2.3 takes up.
1.4AI-native IDEs
Cursor, Claude Code, Devin Desktop, Codex, and Antigravity all put the model inside your project instead of in a separate browser tab. The names change constantly and come from competing companies, but what these tools do has settled into a recognizable shape. They read the files in your project and edit across several of them at once, and depending on the permissions you set, those edits either arrive as diffs you review first or are simply applied. Most importantly, they can run commands: start a dev server, install a package, run the test suite (often one they wrote themselves), and read the error they just caused. Many can now also drive several agents in parallel on separate tasks, and Chapter 3 covers when that's worth doing and when it costs you more than it buys. Because agentic IDEs now both suggest the code and show you whether it works, they're the most powerful vibecodingVibecodingBuilding software by describing intent in natural language and steering an AI toward a working result, instead of hand-typing every line. tool you can pick up, which is why most of this book is about them.
The genuinely new thing is agency, meaning the tool takes actions in your environment, watches what happens, and decides what to do next based on what it just found out. That feedback loop lets it chase down problems on its own, and it's exactly why your verification role gets bigger: a chat assistant that's wrong costs you a bad copy-paste, but an agentic IDE that's wrong can edit ten files and run a destructive command before you've finished reading its original plan.
These tools also give you control over what the model is allowed to see and do, through two separate mechanisms that are easy to confuse. Standing cross-session instructions (Claude Code's memory, Cursor's project rules, Copilot's custom instructions, all one idea under several vendor names) shape what the model tries to do, and ignore rules, the .gitignore-style kind, keep junk out of what it reads. Permissions govern what it is actually allowed to do, which is why most tools let you decide what runs automatically and what needs your approval. A rules file offers guidance where permissions offer a guardrail, and setting both deliberately is the difference between a helpful collaborator and one that confidently and consistently goes off the rails, which is the subject of Chapter 3.
One practical note is that the same model can feel dramatically more or less capable depending on the tool wrapped around it, because the tool decides what the model gets to see, so a disappointing result can be the tool failing to show it the right files rather than the model being incapable of the task.
1.5App builders
Lovable, Bolt, Replit, and v0 sit a layer above the IDE. You describe an app, and they scaffold and often deploy a working version directly, frequently skipping the 'open a code editor' step entirely. v0 specializes in generating polished UI, Lovable and Bolt aim at whole apps, and Replit pairs an agentAgentAn AI system that can take actions, see what happened, and decide what to do next, rather than only producing text. with instant hosting. What they all share is the deliverable: a working app live at a URL.
The speed is the point, since an idea on Monday can be something someone clicks on Tuesday. This is prototyping, and Chapter 12 argues that getting a rough product in front of users early is what actually teaches you whether you're building the right thing, so for validating a concept over a weekend, an app builderApp builderA tool that scaffolds and deploys a working app straight from a description, above the level of a code editor. Optional Chapter M names current examples. beats opening an empty IDE.
The drawback is less control over your code, because the builder makes a thousand small decisions for you, and you inherit a codebase you didn't design and may not fully understand, which can be painful once an app succeeds and has to be maintained. The common claim that builder output is simply unmaintainable overstates it, since one prompt produces a demo, while many careful prompts with real testing produce something more polished. Optional Chapter E, 'Vibe Debugging,' exists precisely because inheriting a builder-generated codebase is now a common real-world starting point.
The three kinds of tools sit on a spectrum. Chat assistants give you words about code, AI-native IDEs give you code in your repo, and app builders give you a deployed product, each step sacrificing control for speed. The right tool depends on the task, so you could use both an IDE and a builder at least once and feel the trade-off yourself instead of taking our word for it.
Tools as a spectrum of abstraction. Moving right trades control over the underlying details for speed and a higher starting point.
1.6ChatGPT wrappers and when a product is just the model
You'll hear people call certain AI products 'ChatGPT wrappers,' and the phrase is blunt on purpose. It means the product is mostly the underlying chat model with a thin layer on top, such as a custom system prompt, a nicer interface, or one workflow button, while the hard work of reasoning and generation still happens inside someone else's API. The label is usually meant as a criticism, and often deserved, though a genuinely well-designed wrapper can still solve a narrow job faster than opening ChatGPT yourself. It's a useful label when you're deciding whether something adds real value or just repackages something that already exists for a monthly fee.
The pattern shows up everywhere in the current tool boom. An 'AI calorie estimator,' a 'resume rewriter,' or a 'study buddy for organic chemistry' is frequently one preset prompt and a form field in front of a general-purpose model, so from the outside it looks like a dedicated product while under the hood it's a single API call with branding. Recognizing ChatGPT wrappers helps you evaluate tools critically before you pay for one.
A wrapper gives you an answer in a chat-shaped box, while a builder gives you something you can deploy, and either can be the right call. They fail differently, since a wrapper fails when the model is wrong or your question was vague, and a builder fails when you inherit scaffolding you can't maintain.
Chapter 12 returns to the business side, where a larger product can swallow a thin wrapper overnight, but for this chapter, the takeaway is simply to know whether you're paying for a model or for someone's packaging of one. If the product is a wrapper, you're paying for convenience and for the fact that someone chose the prompt, designed the interface, and picked the defaults for you. The underlying model is the same one you could use directly, so use a wrapper when the packaging saves you real time, and if you're the one building, ask whether your product is more than packaging before you charge for it.
1.7Top models for vibecoding (a snapshot in time)
Tools change weekly, and models change monthly, so any list of 'the best models' can only be a snapshot rather than a ranking that holds. The table below is dated and describes which models were in common use in AI-native IDEs and app builders when this was written. Before you rely on it, open your tool's model picker and check what the company that makes the model currently publishes [3][5][6], because a newer release, a price cut, or a lack of availability may have changed the answer since this was written. Staying informed is part of the work, and this applies to the whole book: AI moves fast enough that anything specific here may already have been overtaken by the time you read it. You may well be reading this a year or more later, in which case, treat the names as history and the reasoning as current.
For vibecodingVibecodingBuilding software by describing intent in natural language and steering an AI toward a working result, instead of hand-typing every line. specifically, what matters is which model matches the job in front of you inside the tool you're using, which is not always the one that scores highest on benchmarks. Multi-file agentAgentAn AI system that can take actions, see what happened, and decide what to do next, rather than only producing text. work rewards the strongest reasoning model you can afford and actually access, while quick UI polish and copy tweaks might run fine on mid-tier models. Work you can check at a glance and repeat often belongs on the fast, cheap tiers. Optional Chapter M covers install commands, token prices, and effort settings in detail.
Model names in this book are deliberately kept to this section, the tables, and the slides, so that the rest of the text stays useful no matter how far ahead of us you're reading. Everywhere else, the book talks about fast, mid, and frontier tiers, because those categories have been stable for years while the names inside them turn over every few months. Once you understand how the choice works, you can apply it to whatever your tool offers you today.
Start one tier below what you think you need, and move up when the output stops making progress: a wrong architecture, a plan it can't finish, code that doesn't run, instructions it quietly drops, or the moment you realize your prompt left out something it needed. When a task is high-stakes or subtle, running it past another model from a different company is cheap insurance. That catches more than errors, since two models from different labs are less likely to share the same blind spot, which is the move Chapter 2 recommends for hard questions.
Model (provider)
Best for in vibecoding
Typical tool home
Claude Opus 5 (Anthropic)
Multi-file agents, refactors, debugging
Claude Code, Cursor, Devin Desktop
Claude Fable 5 (Anthropic)
The strongest option for the hardest reasoning work
Claude Code, multi-provider IDEs
Claude Sonnet 5 (Anthropic)
Daily edits and medium-complexity features
Claude Code, chat assistants
Claude Haiku 4.5 (Anthropic)
Fast requests: renames, boilerplate, simple fixes
Claude Code, high-volume chat
GPT-5.6 Sol (OpenAI)
Agentic builds inside Codex, OpenAI's coding tool
Codex, Cursor
Gemini 3.5 Flash (Google)
Long-context reads and UI-heavy frontends
Cursor, Antigravity
Models people were reaching for in August 2026. A snapshot, not a permanent ranking. Confirm availability in your own tool before you start.
Check the picker, not the blog post
A model that topped last month's thread may be renamed, repriced, or region-locked today. Your tool's model menu is the source of truth here, not this book.
1.8Frontier vs. fast models
The biggest model isn't automatically the right one, since the choice trades reasoning depth against speed, cost, and what you can actually select today. Settings matter too, and most tools now expose several levels of reasoning effort that can change results as much as switching models does.
Frontier models earn their price when the task needs multi-step reasoning, such as tracing a bug across files or holding a long agentAgentAn AI system that can take actions, see what happened, and decide what to do next, rather than only producing text. plan together without losing the thread. They hallucinate less on subtle logic and hold far more information in view at once, which matters in a long session with a lot of history behind it. They are also better at the thinking that surrounds the code: structuring a messy brief, organizing your ideas, and probing for the gaps and contradictions you left in it. The costs are real, since they take longer, the token bills are higher, and sometimes the depth is simply unnecessary. Newer releases can also arrive with availability restrictions or policy limits [4], so the best model on a leaderboard isn't always one you can select.
Fast and mid-tier models make the opposite trade and struggle when the task needs judgment, so they can pick a plausible library that's wrong for your case, miss a security edge case, or stop remembering what matters once a run gets long. The cost of always reaching for the cheapest model is the time you then spend repairing an architecture that a stronger one would have gotten right.
A useful rule is to match the model to how much can break if the change is wrong. Small, self-contained changes that you can check in a minute belong on the fast tier, while work that touches many files, or where the spec is still vague, or where a mistake would be expensive to unwind, is worth the frontier tier when possible. When in doubt, the mid-tier can be a reasonable place to start.
Tier
Examples (Anthropic, August 2026)
Pros
Cons
Frontier
Fable 5, Opus 5
Deepest reasoning, most context, best for multi-file agents, hard bugs, and vague specs
Slowest and most expensive. The newest models might be restricted or disabled in your picker
Mid
Sonnet 5
Balanced cost and quality for daily edits and medium features
Can struggle on the longest agent runs or the nastiest refactors
Fast
Haiku 4.5
Fastest and cheapest. Ideal for boilerplate, renames, and simple requests
Fills in more of the gaps by guessing, and is weaker on subtle logic and long plans
Pros and cons of frontier vs. fast models. The example names date quickly, though the structure of the trade-off doesn't.
VibecodingAI-native IDEApp builderChatGPT wrapperModel snapshotCode generationHuman-in-the-loopLLMAgencyFrontier model
Key terms
The vocabulary from this chapter, defined.
Vibecoding
Building software by describing intent in natural language and steering an AI toward a working result, instead of hand-typing every line.
Large Language Model (LLM)
A neural network trained on vast amounts of text to predict likely continuations, which is what lets it generate code, prose, and explanations.
AI-native IDE
A code editor where the model can read your files, propose multi-file edits, and run commands directly.
App builder
A tool that scaffolds and deploys a working app straight from a description, above the level of a code editor. Optional Chapter M names current examples.
ChatGPT wrapper
A product that is mostly a chat model with a thin layer on top, such as a custom interface, a preset prompt, or a single workflow, and little else of its own.
Agent
An AI system that can take actions, see what happened, and decide what to do next, rather than only producing text.
Model tier
A rung on a provider's capability ladder. Fast tiers are cheap and quick, while frontier tiers are slow, expensive, and the strongest at reasoning.
Frontier model
The most capable and most expensive model a provider offers, used for the hardest reasoning tasks.
Model snapshot
A list of which models were best for a given task on a given date. Useful for orientation, but check your tool before relying on it.
Human-in-the-loop
A workflow design where a human reviews, corrects, or approves AI output rather than letting the system act unchecked.
Inference
The act of running a trained model to produce an output for a given input. It's what happens every time you send a prompt.
Vibecoding means describing what you want in natural language and steering an AI toward a working result, then verifying that result yourself. We followed the path from autocomplete to agentic IDEs and defined three kinds of tools: chat assistants, AI-native IDEs, and app builders. We named the ChatGPT-wrapper pattern, meaning thin products built on someone else's chat model, and how to tell which kind of tool you're actually paying for. We compared fast, mid, and frontier model tiers and saw why the specific names in any snapshot go stale much faster than the trade-off between them. Then we named the loop this book will keep referring to, which runs from intent through generation and verification to refinement. As tools automate more of the how, your value moves to the what and to deciding whether the result is actually right.
Check your understanding
A few quick questions, then a couple of hands-on exercises.
Q1.In the vibecoding loop (intent, generation, verification, refinement), what's the human's job?
Q2.What was the key shift that GitHub Copilot's 2021 debut represented?
Q3.Which category best describes Cursor and Claude Code?
Q4.A startup sells a '$29/mo AI tutor' that's mostly a general-purpose chat model behind a chemistry-themed system prompt and a chat interface. What term best describes this kind of product?
Q5.Why does this chapter present its 'top models' table with a date on it?
Q6.You need to rename a function across three files with no logic changes. Which choice best balances cost and quality?
Q7.What's the main downside of always defaulting to the smallest, cheapest model?
Practice
Exercise 1.Build a small page that fetches data from a public API and displays the result. Do it once in a chat assistant by copy-pasting code, and once in an AI-native IDE. Write three sentences comparing the friction.
Exercise 2.Name one task from your own experience where you would still rather write the code by hand than describe it to an AI. Justify why.
Exercise 3.Open your AI IDE's model picker and compare it to the snapshot table in Section 1.7. List one model that still matches, one that's new since the table was written, and one task where you would deliberately choose a smaller model.
Exercise 4.Run the same small task twice in your IDE, once on your tool's fast tier and once on its frontier tier. Write two sentences comparing the two models on speed, on whether the price difference was noticeable, and on whether you could tell the two results apart at all.
Vibecoding: Introduction to Applied AI
A practical guide to making real software with AI.
Eli Young, Andy Luu, and Michael Wehar
This is a working draft. Author order has not been settled yet, and the names appear in no particular order.
Free for classroom and personal use under CC BY-NC 4.0.
Acknowledgments
We wrote this book to teach people how to vibecode, because we could not find an existing textbook that did. This is the first edition, the 2027 edition. If you have comments or concerns, we would like to hear about them.
Hiding AI assistance would be odd in a book about vibecoding. Anthropic models, accessed with Claude Code, drafted portions that we read, evaluated, and edited. We take full responsibility for the content.
1. Introduction to Vibecoding and the AI Ecosystem
What vibecoding is, where it came from, and the tools people use to do it.
Vibecoding means building software by describing what you want in natural language and letting an AI produce the code. In practice, the word covers a wide range. Some people fire off one prompt and ship whatever comes back. Others specify carefully and check every result. This book teaches the second kind, and this chapter maps the tools and the vocabulary you'll need for it.
By the end of this chapter, you will be able to
Define vibecoding and explain its four-step loop: intent, generation, verification, refinement
Summarize the major milestones in AI-assisted programming, from autocomplete to agentic IDEs
Tell chat assistants, AI-native IDEs, and app builders apart, and know when to use each
Recognize a ChatGPT wrapper and determine whether it adds real value beyond the base model
Choose a model that fits the job, from fast and cheap to slow and powerful, and know when to check whether better options have shipped
Compare pros and cons of frontier versus fast model tiers and choose appropriately for a task
Explain why human verification becomes more important as AI tools gain autonomy
1.1 What's vibecoding?
Vibecoding is a workflow where you say what you want in plain language, direct an AI model to write the code, and check, correct, and iterate until it actually does what you intended. The term was popularized in February 2025, when Andrej Karpathy, a founding member of OpenAI and former head of AI at Tesla, described 'fully giving in to the vibes' and forgetting that the code even exists [1]. It spread because it named something a lot of people had quietly started doing: building by conversation.
The reason vibecoding spread is speed. Even an engineer who could write the same code by hand can't do so as fast as a model, so a draft that would have taken an hour arrives in minutes flat. The deeper win comes next, because while the model works, you're free to write the next prompt or steer a second agent on another task. Your output is no longer limited by how fast you produce code, but by how fast you can decide what to build and confirm it was built right.
The careless end of that range is what gives vibecoding its reputation, and the difference between the two ends is the difference between a demo and a product. This book is aimed at the product end, which means you stay responsible for the result while the AI produces it quickly, tirelessly, and sometimes wrongly. A demo only has to work once, in front of people rooting for it. A product has to work at 3 a.m. for a stranger who isn't.
You'll sometimes hear 'vibecoder' used as an insult, shorthand for someone who can't really program and just pastes whatever the model says. Reject that framing, because every generation of tooling has drawn this line and then witnessed it move: assembly programmers doubted compilers, and C programmers doubted garbage collection. The question is whether the work holds up and whether the person who shipped it can stand behind it. A vibecoder who specifies clearly and verifies carefully is doing real engineering, which is the standard this book holds you to.
None of this means programming knowledge stops mattering, only that it shows up differently. You still need to recognize what 'correct' looks like and why an off-by-one breaks a loop, because those are the kinds of judgments you'll constantly be making about the model's output. What changes is where your attention goes, moving away from recalling syntax and toward specifying and verifying.
You plan and architect the build, you direct and steer the model as it generates the code, you check that the result holds to your intent, and after that check you draft your next request for code before the cycle repeats.
The four-step vibecoding loop, in which you plan and architect the build, direct and steer generation, verify the result against your intent, and after verification draft the next request for code before the cycle repeats.
1.2 A short history of AI-assisted programming
AI assistance in code editors didn't arrive all at once, and seeing the steps makes today's tools less mysterious, starting with the autocomplete and IntelliSense that shipped in Visual Studio back in 1997. These were rule-based and statistical suggestions scoped to a single identifier or line, driven by the symbols already defined in your project. Useful, but they only ever finished a thought you had already started typing.
The next wave was search, meaning Google and Stack Overflow, where the human still did most of the reasoning manually, while the machine only did retrieval, until the real change came in 2021 with GitHub Copilot. It was the first widely used tool to put a large language model (LLM), a network trained on vast amounts of public code, directly in the editor, suggesting whole multi-line blocks from only a comment or a function name.
From there, chat-based assistants (ChatGPT in late 2022, Claude in early 2023) could hold a conversation about an entire problem, explain their reasoning, and adapt across many turns. Then agentic IDEs (Cursor, Claude Code, Devin Desktop) narrowed the gap further by letting the model read, edit, and run code across many files on its own. Given permission, these tools reach past the editor and onto your machine, which turned 'suggest some text' into 'do the task.'
The pattern is worth naming because it predicts what comes next. Each wave automated more of the how and left the human more of the what and the question of whether the result is right, and vibecoding is the current step in that progression. Specifying and verifying are the skills that every wave so far has left in human hands.
1.3 Chat assistants
ChatGPT, Claude, and Gemini are general-purpose chat assistants. You should keep in mind that the product is separate from the model within it. For example, ChatGPT is the product, while the LLM is only one part of a package that also includes the interface, the saved instructions, and any tools it can access. These assistants answer questions, explain code, brainstorm designs, and write snippets. Most can now run code in a sandbox of their own, and some can reach your files if you connect them. What stays true is that, by default, they work on whatever you paste in, or on what they have produced themselves in that conversation, not on your project on your machine. That boundary is useful because it makes a chat assistant a low-stakes place to think before you commit to anything.
So the best use of a chat assistant is often before you write any code. Pressure-test an approach, compare two code libraries, or ask it to poke holes in your plan. It's a better thinking partner for the what and the why than a code generator for the how, which is what the IDE-level tools do better anyway.
Each of these products is really a family of models at different tiers of capability and cost [3][5][6], from a fast, cheap one for easy requests up to a larger, slower one for hard reasoning. Using the most expensive model to change a button color could waste money, while using the cheapest to design a database schema could waste your afternoon fixing its mistakes.
One limit is worth stating up front. A chat assistant only knows what's in the conversation plus what it learned during training, which stopped at a fixed date. Ask it about a library version released last month, and it will often answer confidently about the version it remembers. Most assistants can now search the web, which helps a great deal with anything recent, though it introduces a failure of its own, because a model can't reliably tell a good source from a bad one and will summarize either with the same confidence. This is a first taste of the hallucination problem that Chapter 9 makes central, and of the context questions that Section 2.3 takes up.
1.4 AI-native IDEs
Cursor, Claude Code, Devin Desktop, Codex, and Antigravity all put the model inside your project instead of in a separate browser tab. The names change constantly and come from competing companies, but what these tools do has settled into a recognizable shape. They read the files in your project and edit across several of them at once, and depending on the permissions you set, those edits either arrive as diffs you review first or are simply applied. Most importantly, they can run commands: start a dev server, install a package, run the test suite (often one they wrote themselves), and read the error they just caused. Many can now also drive several agents in parallel on separate tasks, and Chapter 3 covers when that's worth doing and when it costs you more than it buys. Because agentic IDEs now both suggest the code and show you whether it works, they're the most powerful vibecoding tool you can pick up, which is why most of this book is about them.
The genuinely new thing is agency, meaning the tool takes actions in your environment, watches what happens, and decides what to do next based on what it just found out. That feedback loop lets it chase down problems on its own, and it's exactly why your verification role gets bigger: a chat assistant that's wrong costs you a bad copy-paste, but an agentic IDE that's wrong can edit ten files and run a destructive command before you've finished reading its original plan.
These tools also give you control over what the model is allowed to see and do, through two separate mechanisms that are easy to confuse. Standing cross-session instructions (Claude Code's memory, Cursor's project rules, Copilot's custom instructions, all one idea under several vendor names) shape what the model tries to do, and ignore rules, the .gitignore-style kind, keep junk out of what it reads. Permissions govern what it is actually allowed to do, which is why most tools let you decide what runs automatically and what needs your approval. A rules file offers guidance where permissions offer a guardrail, and setting both deliberately is the difference between a helpful collaborator and one that confidently and consistently goes off the rails, which is the subject of Chapter 3.
One practical note is that the same model can feel dramatically more or less capable depending on the tool wrapped around it, because the tool decides what the model gets to see, so a disappointing result can be the tool failing to show it the right files rather than the model being incapable of the task.
1.5 App builders
Lovable, Bolt, Replit, and v0 sit a layer above the IDE. You describe an app, and they scaffold and often deploy a working version directly, frequently skipping the 'open a code editor' step entirely. v0 specializes in generating polished UI, Lovable and Bolt aim at whole apps, and Replit pairs an agent with instant hosting. What they all share is the deliverable: a working app live at a URL.
The speed is the point, since an idea on Monday can be something someone clicks on Tuesday. This is prototyping, and Chapter 12 argues that getting a rough product in front of users early is what actually teaches you whether you're building the right thing, so for validating a concept over a weekend, an app builder beats opening an empty IDE.
The drawback is less control over your code, because the builder makes a thousand small decisions for you, and you inherit a codebase you didn't design and may not fully understand, which can be painful once an app succeeds and has to be maintained. The common claim that builder output is simply unmaintainable overstates it, since one prompt produces a demo, while many careful prompts with real testing produce something more polished. Optional Chapter E, 'Vibe Debugging,' exists precisely because inheriting a builder-generated codebase is now a common real-world starting point.
The three kinds of tools sit on a spectrum. Chat assistants give you words about code, AI-native IDEs give you code in your repo, and app builders give you a deployed product, each step sacrificing control for speed. The right tool depends on the task, so you could use both an IDE and a builder at least once and feel the trade-off yourself instead of taking our word for it.
Tools as a spectrum of abstraction. Moving right trades control over the underlying details for speed and a higher starting point.
1.6 ChatGPT wrappers and when a product is just the model
You'll hear people call certain AI products 'ChatGPT wrappers,' and the phrase is blunt on purpose. It means the product is mostly the underlying chat model with a thin layer on top, such as a custom system prompt, a nicer interface, or one workflow button, while the hard work of reasoning and generation still happens inside someone else's API. The label is usually meant as a criticism, and often deserved, though a genuinely well-designed wrapper can still solve a narrow job faster than opening ChatGPT yourself. It's a useful label when you're deciding whether something adds real value or just repackages something that already exists for a monthly fee.
The pattern shows up everywhere in the current tool boom. An 'AI calorie estimator,' a 'resume rewriter,' or a 'study buddy for organic chemistry' is frequently one preset prompt and a form field in front of a general-purpose model, so from the outside it looks like a dedicated product while under the hood it's a single API call with branding. Recognizing ChatGPT wrappers helps you evaluate tools critically before you pay for one.
A wrapper gives you an answer in a chat-shaped box, while a builder gives you something you can deploy, and either can be the right call. They fail differently, since a wrapper fails when the model is wrong or your question was vague, and a builder fails when you inherit scaffolding you can't maintain.
Chapter 12 returns to the business side, where a larger product can swallow a thin wrapper overnight, but for this chapter, the takeaway is simply to know whether you're paying for a model or for someone's packaging of one. If the product is a wrapper, you're paying for convenience and for the fact that someone chose the prompt, designed the interface, and picked the defaults for you. The underlying model is the same one you could use directly, so use a wrapper when the packaging saves you real time, and if you're the one building, ask whether your product is more than packaging before you charge for it.
1.7 Top models for vibecoding (a snapshot in time)
Tools change weekly, and models change monthly, so any list of 'the best models' can only be a snapshot rather than a ranking that holds. The table below is dated and describes which models were in common use in AI-native IDEs and app builders when this was written. Before you rely on it, open your tool's model picker and check what the company that makes the model currently publishes [3][5][6], because a newer release, a price cut, or a lack of availability may have changed the answer since this was written. Staying informed is part of the work, and this applies to the whole book: AI moves fast enough that anything specific here may already have been overtaken by the time you read it. You may well be reading this a year or more later, in which case, treat the names as history and the reasoning as current.
For vibecoding specifically, what matters is which model matches the job in front of you inside the tool you're using, which is not always the one that scores highest on benchmarks. Multi-file agent work rewards the strongest reasoning model you can afford and actually access, while quick UI polish and copy tweaks might run fine on mid-tier models. Work you can check at a glance and repeat often belongs on the fast, cheap tiers. Optional Chapter M covers install commands, token prices, and effort settings in detail.
Model names in this book are deliberately kept to this section, the tables, and the slides, so that the rest of the text stays useful no matter how far ahead of us you're reading. Everywhere else, the book talks about fast, mid, and frontier tiers, because those categories have been stable for years while the names inside them turn over every few months. Once you understand how the choice works, you can apply it to whatever your tool offers you today.
Start one tier below what you think you need, and move up when the output stops making progress: a wrong architecture, a plan it can't finish, code that doesn't run, instructions it quietly drops, or the moment you realize your prompt left out something it needed. When a task is high-stakes or subtle, running it past another model from a different company is cheap insurance. That catches more than errors, since two models from different labs are less likely to share the same blind spot, which is the move Chapter 2 recommends for hard questions.
Model (provider)
Best for in vibecoding
Typical tool home
Claude Opus 5 (Anthropic)
Multi-file agents, refactors, debugging
Claude Code, Cursor, Devin Desktop
Claude Fable 5 (Anthropic)
The strongest option for the hardest reasoning work
Claude Code, multi-provider IDEs
Claude Sonnet 5 (Anthropic)
Daily edits and medium-complexity features
Claude Code, chat assistants
Claude Haiku 4.5 (Anthropic)
Fast requests: renames, boilerplate, simple fixes
Claude Code, high-volume chat
GPT-5.6 Sol (OpenAI)
Agentic builds inside Codex, OpenAI's coding tool
Codex, Cursor
Gemini 3.5 Flash (Google)
Long-context reads and UI-heavy frontends
Cursor, Antigravity
Check the picker, not the blog postA model that topped last month's thread may be renamed, repriced, or region-locked today. Your tool's model menu is the source of truth here, not this book.
1.8 Frontier vs. fast models
The biggest model isn't automatically the right one, since the choice trades reasoning depth against speed, cost, and what you can actually select today. Settings matter too, and most tools now expose several levels of reasoning effort that can change results as much as switching models does.
Frontier models earn their price when the task needs multi-step reasoning, such as tracing a bug across files or holding a long agent plan together without losing the thread. They hallucinate less on subtle logic and hold far more information in view at once, which matters in a long session with a lot of history behind it. They are also better at the thinking that surrounds the code: structuring a messy brief, organizing your ideas, and probing for the gaps and contradictions you left in it. The costs are real, since they take longer, the token bills are higher, and sometimes the depth is simply unnecessary. Newer releases can also arrive with availability restrictions or policy limits [4], so the best model on a leaderboard isn't always one you can select.
Fast and mid-tier models make the opposite trade and struggle when the task needs judgment, so they can pick a plausible library that's wrong for your case, miss a security edge case, or stop remembering what matters once a run gets long. The cost of always reaching for the cheapest model is the time you then spend repairing an architecture that a stronger one would have gotten right.
A useful rule is to match the model to how much can break if the change is wrong. Small, self-contained changes that you can check in a minute belong on the fast tier, while work that touches many files, or where the spec is still vague, or where a mistake would be expensive to unwind, is worth the frontier tier when possible. When in doubt, the mid-tier can be a reasonable place to start.
Tier
Examples (Anthropic, August 2026)
Pros
Cons
Frontier
Fable 5, Opus 5
Deepest reasoning, most context, best for multi-file agents, hard bugs, and vague specs
Slowest and most expensive. The newest models might be restricted or disabled in your picker
Mid
Sonnet 5
Balanced cost and quality for daily edits and medium features
Can struggle on the longest agent runs or the nastiest refactors
Fast
Haiku 4.5
Fastest and cheapest. Ideal for boilerplate, renames, and simple requests
Fills in more of the gaps by guessing, and is weaker on subtle logic and long plans
Summary
Vibecoding means describing what you want in natural language and steering an AI toward a working result, then verifying that result yourself. We followed the path from autocomplete to agentic IDEs and defined three kinds of tools: chat assistants, AI-native IDEs, and app builders. We named the ChatGPT-wrapper pattern, meaning thin products built on someone else's chat model, and how to tell which kind of tool you're actually paying for. We compared fast, mid, and frontier model tiers and saw why the specific names in any snapshot go stale much faster than the trade-off between them. Then we named the loop this book will keep referring to, which runs from intent through generation and verification to refinement. As tools automate more of the how, your value moves to the what and to deciding whether the result is actually right.
Key terms
Vibecoding. Building software by describing intent in natural language and steering an AI toward a working result, instead of hand-typing every line.
Large Language Model (LLM). A neural network trained on vast amounts of text to predict likely continuations, which is what lets it generate code, prose, and explanations.
AI-native IDE. A code editor where the model can read your files, propose multi-file edits, and run commands directly.
App builder. A tool that scaffolds and deploys a working app straight from a description, above the level of a code editor. Optional Chapter M names current examples.
ChatGPT wrapper. A product that is mostly a chat model with a thin layer on top, such as a custom interface, a preset prompt, or a single workflow, and little else of its own.
Agent. An AI system that can take actions, see what happened, and decide what to do next, rather than only producing text.
Model tier. A rung on a provider's capability ladder. Fast tiers are cheap and quick, while frontier tiers are slow, expensive, and the strongest at reasoning.
Frontier model. The most capable and most expensive model a provider offers, used for the hardest reasoning tasks.
Model snapshot. A list of which models were best for a given task on a given date. Useful for orientation, but check your tool before relying on it.
Human-in-the-loop. A workflow design where a human reviews, corrects, or approves AI output rather than letting the system act unchecked.
Inference. The act of running a trained model to produce an output for a given input. It's what happens every time you send a prompt.
Check your understanding
Q1. In the vibecoding loop (intent, generation, verification, refinement), what's the human's job?
Specify the intent clearly, then verify the result and correct whatever came back wrong
Type every line of code by hand, using the model only for reference
Wait for the model to verify its own code before reviewing anything
Refine the code only after an outside reviewer approves the model's plan
Answer: A. The model produces the drafts. Specifying what you want and deciding whether you got it stay with you. That division of labor is the whole point of the loop.
Q2. What was the key shift that GitHub Copilot's 2021 debut represented?
It was the first appearance of code autocomplete in any editor
It was the first editor tool that could run your code automatically
It was the first product to deploy an app straight from a description
It moved from single-line suggestions to model-generated multi-line blocks
Answer: D. Autocomplete and IntelliSense existed long before Copilot, going back to 1997. Copilot's change was using a large language model to generate plausible multi-line completions instead of single-line suggestions.
Q3. Which category best describes Cursor and Claude Code?
Chat assistants, because you talk to them in a conversation
App builders, because they deploy a working product for you
AI-native IDEs, because they read your files and run commands
Search engines, because they retrieve code from public sources
Answer: C. They live inside your project, read many files for context, and can execute commands. That last capability is what separates an AI-native IDE from a chat assistant or an app builder.
Q4. A startup sells a '$29/mo AI tutor' that's mostly a general-purpose chat model behind a chemistry-themed system prompt and a chat interface. What term best describes this kind of product?
An AI-native IDE, because it's built on a language model
A ChatGPT wrapper, because it's a preset prompt over a chat API
A frontier model, because it runs on a large general-purpose model
A supervised workflow, because a human tutor reviews the output
Answer: B. The product's core is a chat API plus a thin preset layer, which is the wrapper pattern, and it isn't an IDE because it has no access to your repository.
Q5. Why does this chapter present its 'top models' table with a date on it?
Model names, prices, and availability change often, so any list is a snapshot
Model rankings become permanent once they're published in a textbook
Only one company makes models that are suitable for vibecoding work
The date is included for copyright reasons and not for technical accuracy
Answer: A. Model families move fast, so a dated snapshot can orient you today, while your own tool's picker is the source of truth tomorrow.
Q6. You need to rename a function across three files with no logic changes. Which choice best balances cost and quality?
The newest frontier model, because it's always the most accurate option
A frontier model, because only frontier models can edit code across files
The fast tier, because the task is small, checkable, and little can break
No model handles renames reliably, so you should edit the files by hand
Answer: C. Renames across a few files break little if they go wrong and take a moment to check, so a small, cheap model handles them well. A frontier model would work, but it costs more for no meaningful gain.
Q7. What's the main downside of always defaulting to the smallest, cheapest model?
It refuses to answer coding questions that involve more than one file
It may guess wrong on subtle logic, costing more repair time than it saved
It can't access the internet, so it never has any current information
It's only available inside chat assistants and never inside an IDE
Answer: B. Fast tiers save money on quick, easy requests but fall down on judgment-heavy work. The cost you might not see coming is the repair time after a cheap model picks an architecture that looks reasonable but is wrong.
Practice
Exercise 1. Build a small page that fetches data from a public API and displays the result. Do it once in a chat assistant by copy-pasting code, and once in an AI-native IDE. Write three sentences comparing the friction. (Hint: Count the manual copy, paste, and file-creation steps each approach needed, and note how you found out whether the code actually ran. The multi-file, one-dependency shape is what makes the difference visible.)
Exercise 2. Name one task from your own experience where you would still rather write the code by hand than describe it to an AI. Justify why. (Hint: Think about tasks where the specification is harder to write in English than the code itself, like precise bit manipulation or an exact numerical formula you already know.)
Exercise 3. Open your AI IDE's model picker and compare it to the snapshot table in Section 1.7. List one model that still matches, one that's new since the table was written, and one task where you would deliberately choose a smaller model. (Hint: If your picker already disagrees with the table, note exactly what changed. The gap between the two is the point of the exercise.)
Exercise 4. Run the same small task twice in your IDE, once on your tool's fast tier and once on its frontier tier. Write two sentences comparing the two models on speed, on whether the price difference was noticeable, and on whether you could tell the two results apart at all. (Hint: If you couldn't tell the difference, you just found a task to keep on the fast tier permanently. If the frontier run handled something the fast one got wrong, you found where the extra depth pays for itself.)
Sources
Andrej Karpathy, the original "vibe coding" post (X, Feb 2, 2025) (https://x.com/karpathy/status/1886192184808149383)
Model Context Protocol, the open standard for connecting AI tools to external data and tools, now a Linux Foundation project (https://modelcontextprotocol.io)
Claude models overview: current lineup, context windows, and pricing (https://platform.claude.com/docs/en/about-claude/models/overview)
Claude Mythos 5 and Project Glasswing: invitation-only access for defensive cybersecurity work (https://anthropic.com/glasswing)
Google Gemini models documentation (https://ai.google.dev/gemini-api/docs/models)
2. Prompting and Specification Writing
Turning a rough idea into a specification that a model can actually execute.
How you write a prompt determines the quality of what the AI gives back, so this chapter treats prompting as engineering. You gather requirements, write a product requirement document, supply the right context, and refine in small steps instead of starting over.
By the end of this chapter, you will be able to
Structure a prompt with role, task, constraints, and output format
Write a short product requirement document that states must-haves, non-goals, and acceptance criteria
Apply context engineering to keep a model's input focused and relevant
Refine AI output step by step, using changes small enough to check one at a time
Explain why identical prompts can produce different outputs and what to do about it
2.1 Prompt engineering fundamentals
A good prompt is structured, and prompt engineering comprises four components that usually do more for you than clever wording. First, a role, telling the model what perspective to write from, such as 'you're a senior backend engineer who values simplicity.' Second, a task, the thing you want done, such as 'add rate limiting to this endpoint.' Third, a set of constraints, the rules it has to work inside, such as 'use the existing Redis client, add no new dependencies, keep it under 40 lines.' Fourth, an output format, saying what you want back, such as 'show only the changed function.' None of these are magic words, just the information the model needs, and anything you leave out is something it has to guess.
A language model fills every gap you leave with the statistically most likely choice, which may not be the choice you had in mind, and this is also why generic prompts produce generic output: the less you specify, the closer the result sits to the average of everything the model was trained on. Ask for 'a function to validate emails,' and you'll get one, but the details are now the model's to decide. Whether it accepts an address like name+tag@example.com, how it handles non-English characters, and whether it returns true or false versus raising an error are all decisions the model made because you didn't, which is how a vague prompt produces confident code that isn't what you intended. Designers have known this failure mode for decades. No phrase has sunk more redesigns than 'make it pop.'
Most of the time, you just describe the task and send it, which is called zero-shot prompting because you give the model no examples to work from [1], and it works fine for most things, though two techniques help when it doesn't. Few-shot prompting means showing the model a small number of worked examples of the input and the output you want, rather than only describing them. Models copy the pattern in a worked example far more reliably than they follow a written description of it [1]. Chain-of-thought prompting means asking the model to work through its reasoning step by step before committing to an answer, which measurably improves accuracy on multi-stage problems, partly because it can get each stage right in turn and partly because a plan written out is a plan you can correct before it becomes code [2]. Adding 'let's think step by step' is often enough to trigger it [3].
Your instructions can live in two places. The system prompt holds overarching rules for the whole session, such as 'explain things to someone who hasn't used Rust before' or 'never commit our .env to GitHub,' which you set once and expect to hold, while your individual messages carry the task of the moment. This is the same idea as the rules files in Section 1.4, applied within a single conversation. Putting your standing preferences in the system prompt, or in a rules file as Chapter 3 covers, saves you from retyping them every message and stops the model from forgetting them three turns later [4].
The anatomy of a usable spec. Most weak prompts are missing one of these four layers, usually the acceptance criteria.
The same request, vague versus specific. The specific version leaves the model far less room to guess wrong.
The four-part checkBefore sending any non-trivial prompt, check that it names a role, a task, the constraints, and the output format you want. Most 'the AI ignored what I wanted' moments are really one of those four left unsaid.
2.2 Requirement gathering and PRDs
Before you prompt, describe the requirements and decide what 'done' means, because otherwise the model decides for you. A product requirement document, or PRD, forces that decision early, and even a one-page version is enough as long as it's explicit rather than formal.
A useful PRD has five parts: the user it's for, the problem in one sentence, the must-have behaviors, the non-goals (the things you're deliberately not building in this version), and the acceptance criteria, meaning the concrete conditions you can check to prove it works. Non-goals do surprising amounts of work here, since writing 'this doesn't handle teams, only individual users' might prevent the model from helpfully scaffolding a permissions system you never wanted.
The payoff is that the PRD and the prompt become the same document, so the hard thinking you do once kills two birds with one stone. Those five parts already contain almost everything the model needs, so you can paste the document in with very little rewriting.
Researchers at Salesforce AI Research and Microsoft ran more than 200,000 simulated conversations across a range of leading models and found that when a task's requirements were revealed one turn at a time instead of stated upfront, performance dropped by an average of 39 percent across six generation tasks [8]. The models made early assumptions about the unstated parts, committed to them, and didn't recover when the real requirement finally arrived. Any requirement you only discover halfway through is one that the model has already guessed at and built around. You pay once to undo the guess and again to build what you actually wanted. None of this means a PRD guarantees a good result, since a clear spec can still produce the wrong thing, but stating the requirements up front measurably reduces how much you have to undo [8].
Brain-dump, then let it interview youYou don't have to write the PRD cold. Brain-dump first, typing your vision fast and messy in whatever order it comes out, then send it and ask the model to interview you. It can identify the ambiguities for you before any code starts getting produced. Turning a rough dump into a real spec through that back-and-forth is usually faster and cheaper than drafting a clean PRD from a blank page, because a model questioning your plan will raise things you might not have raised on your own. Define the spec before any building starts, since a rambling build conversation costs far more tokens than one prompt that already says what you want.
2.3 Context engineering
Context is everything the model can see when it answers you, meaning the files you showed it, the examples you gave, the rules you set, and the conversation so far. Context engineering means choosing what goes in there on purpose and choosing what to leave out, which has become one of the most valuable skills in practical AI work. The research on long inputs is consistent on one point, which is that what you put in the chat changes the answer at least as much as which model you picked [6][7].
The part that surprises people is that more context isn't better, since pasting more code than needed thins the model's attention until it can't tell which of the twenty files matters, a problem Chapter 4 names context rot and explains in full.
The worst clutter isn't the obviously unrelated kind, which models mostly ignore, but the material that looks current and isn't, such as an old version of a schema or a decision you already reversed. Research on distraction finds that on-topic material pulls a model off course the most [9], so when you trim, trim for what's stale or contradictory first and worry about length second.
So include what the model needs and nothing else: the function you're modifying, the type of data moving through it (say, integers representing user scores), an example of the pattern to follow (how the nearby functions format what they return), and any rule the code itself doesn't show (that a score must never be negative).
Context also decays over a long session, because the instructions you gave at the start get buried under thousands of newer tokens and lose their influence, and some tools eventually summarize or drop them entirely [7]. Switching the same session to an unrelated task has the same effect, since the old work stays in the window and competes with the new. Most coding tools expose a compaction command that summarizes the history and continues from it. Claude Code's /clear goes further and starts a fresh conversation with an empty context, without making you leave the terminal session you're already in [10]. Either one beats adding to a thread that the model is already half-forgetting.
2.4 Iterative refinement
Work in small changes. A diff is the set of lines a change adds, removes, or edits, with removals in red and additions in green, and your tool can put one in front of you before you accept anything, so ask for one focused change, then run it and check that it did what you asked before you accept it. Move on to the next change only once the current one does what you wanted, which feels slower but usually finishes sooner, because asking for a whole feature at once produces a diff big enough to bury bugs that will be hard to find.
Ask for small steps while you're still working out what you want, and hand the model bigger pieces of work only once it's clearly following your pattern. You'll develop a feel for this, and when in doubt, go smaller, since being too careful could cost you a few minutes, while being too eager could cost you the afternoon.
When a change is wrong, being specific is what gets it fixed. A prompt that names the exact issue, such as an error thrown on an empty array or a total off by one cent on refunds, narrows the model toward the actual fix, and pasting the error text itself is even better, while vagueness tends to produce a different version of the same mistake. If you can't describe the issue precisely, ask the model to add logging, then reproduce the issue and feed the output back, since letting it see the actual values beats guessing at them together.
One more habit separates smooth sessions from frustrating ones. When the model goes badly off track, revert to an earlier checkpoint and start again with clearer direction instead of patching a flawed result, since most coding tools keep a history you can roll back to, and version control (Optional Chapter P) does the same for the files. A conversation carrying three wrong turns carries that confusion into everything after it, and this is measured: once a model takes a wrong turn, it tends to stay lost, and the damage shows up as unreliability more than lost ability, so the same session can swing between success and failure [8]. The practical response is to start a new conversation, and before you retry, gather the requirements now scattered across a dozen turns into one clean prompt, which you can do by asking the model to summarize everything you've specified so far and carrying that summary across.
Section 1.7 suggested asking a second model when a task is high-stakes, and it works because two models from different companies rarely share the same blind spot. Agreement is mild evidence you're right, and disagreement tells you to slow down and verify properly.
2.5 Why the same prompt never produces the same output twice
Try this once, and you'll never forget it. Open a blank chat, type 'build me a landing page for a coffee shop,' and run it, then paste the exact same words into an empty chat and run it again. Layout, wording, and choice of components will all differ between the two pages, and a third run gives you a third version, which is how a language model behaves and says nothing about whether your tool is working.
When a model generates text, it picks the next word from a range of possible next words, each with a different probability, then picks again and again until the response is finished. How much of that range it will consider is governed by a setting called temperature, where a low value keeps it close to the safest, most predictable choice and a high value lets less likely options through. In most coding tools, you never touch this yourself, since the provider or the tool sets it for you, but it's worth knowing it exists because it explains the variation you see. Even at the lowest temperature, two runs are rarely identical, both because generation is probabilistic by nature and because changes you might not have noticed, such as a model version update or how your request was routed, can send it down a different path [5].
App builders and agents add a second layer of variation because they don't call the model just once. They plan, choose libraries, create files, and make dozens of small decisions you never see, so two runs of the same one-line prompt can come back structured in fundamentally different ways you never anticipated.
The practical takeaway is that a prompt alone won't reliably reproduce a result, so write down what 'good enough' means, keep the parts that matter in a PRD or a rules file, and check what you actually got. If you need the same output twice, reuse the files the model produced the first time.
The three-run habitBefore you trust a one-shot build, run the same prompt twice more in blank sessions, so you can see which parts of the result were actually specified and which the model chose for you. Wherever the three disagree is a decision you cared about but never stated, and that belongs in the spec before you build on it. Those two extra runs can go on a cheaper tier, since they exist to find the gaps in your prompt, and you'll likely throw the builds away.
Summary
How you prompt determines what you get back. We treated prompting as engineering: four components in a request, a messy brain-dump the model interviews you about until it becomes a PRD that defines 'done,' context chosen deliberately instead of dumped in, and refinement in small changes you can check as you go. Identical prompts still produce different results across runs, which is why the written spec is what you rely on rather than the prompt.
Key terms
Prompt engineering. Structuring a request so it reliably produces the desired output.
System prompt. Instructions that apply to a whole conversation and are sent with every turn, reiterating the rules you want the model to keep following.
Zero-shot. A prompt that asks a model to do a task with no examples, just the instruction.
Few-shot. A prompt that gives the model one or more examples of the input and output you want, so it can match the pattern.
Chain-of-thought. Prompting a model to reason step by step before giving a final answer, which improves accuracy on multi-step problems.
PRD (Product Requirement Document). A short spec naming the user, the problem, must-haves, non-goals, and acceptance criteria.
Acceptance criteria. The specific conditions you can check to prove a feature works. If you cannot test it, it is not an acceptance criterion.
Non-goals. Things you decide up front not to build, so the model does not build them for you.
Context. Everything the model can see when it answers: your files, your examples, your rules, and the conversation so far.
Context engineering. Deliberately choosing what the model sees and what to leave out.
Context rot. The failure that sets in when useful information gets buried under so much irrelevant material that the model loses track of what matters.
Temperature. A setting that controls how adventurous a model's word choices are. Low keeps it predictable. High makes it more varied.
Sampling. The process by which a model picks each next word from a range of possibilities instead of retrieving one fixed answer.
Diff. The set of lines a change adds, removes, or edits. It's what your tool shows you before you accept an edit.
Cross-checking models. Running the same prompt through a second, independent model to catch mistakes or blind spots that the first one might share with your own assumptions.
Check your understanding
Q1. Why can a vague prompt produce confident code that's wrong?
The model refuses to answer a prompt it considers too vague
A vague prompt will always exceed the model's context window
The model fills any gap you leave with its own assumption
The API rejects prompts that don't specify an output format
Answer: C. Every unstated detail is a decision point. Leave it unstated, and the model picks something plausible, which may not be what you actually wanted.
Q2. What's the purpose of explicitly listing 'non-goals' in a PRD?
To state what you're deliberately not building, which scopes the work
To make the document long enough to be taken seriously
To list every feature you might want to add in a future version
Non-goals are a design-review artifact and not part of a PRD
Answer: A. Writing down what you aren't building keeps the work scoped and stops the model from adding features you never asked for.
Q3. Why is asking for one focused change at a time generally better than one giant request?
Large requests are rejected by the model once they pass a size limit
It uses fewer tokens overall, so the same work costs you less
It isn't better, and larger requests finish the same work in less time
Smaller diffs are easier to check, so a bug is easier to find
Answer: D. Checking gets harder as a diff grows. Small, focused changes keep each step something you can actually verify.
Q4. You paste the exact same prompt, 'make a blue website,' into three blank chats and get three different layouts. What's the most accurate explanation?
The model is malfunctioning and should be returning identical results
The model picks each word from a range of possible options, so two runs differ even on identical input
Each separate chat is served by a completely different underlying model
Identical prompts only differ if the tool has been misconfigured somewhere
Answer: B. Models generate word by word, choosing from a range of possibilities each time. Identical input doesn't guarantee identical output, and a low temperature only reduces the variation instead of removing it.
Q5. According to the chapter, what makes context clutter most dangerous to a model's output?
Clearly unrelated material, which is what models struggle with most
Any single file longer than about fifty lines, regardless of content
Old material that still looks current, like a schema you already replaced
Clutter has no measurable effect, because models weigh all input equally
Answer: C. Research on distraction finds that on-topic material pulls a model off course the most. Something stale that still looks current is more dangerous than something obviously irrelevant, which models mostly ignore.
Practice
Exercise 1. Write a one-page PRD (user, problem, must-haves, non-goals, acceptance criteria) for a simple habit tracker. Then paste that PRD in as your prompt, almost word for word, and see what the model builds. (Hint: If the output violates a non-goal you listed, that's useful information, so catch it and correct it. Notice how much of the prompt you didn't have to write, because the PRD had already said it.)
Exercise 2. Take a vague one-sentence prompt you've used before, like 'build me a todo app,' and rewrite it with a role, a task, constraints, and an output format. If you've never written one, invent a task you actually care about and start there. Compare the two outputs. If the structured version is still not what you wanted, write a third prompt, paste it into a completely new chat, and note what you had left out. (Hint: Keep both outputs and mark every place the vague version guessed at something your structured version specified.)
Exercise 3. In three blank chats, run the exact same short prompt, for example, 'make a simple blue website.' Screenshot or list three concrete differences, naming the specific choice behind each one: layout, wording, components. (Hint: If all three agree on something you never specified, you found a hidden default. If they disagree, you found a gap in your spec that you should have written down.)
Sources
Brown et al., Language Models are Few-Shot Learners, the GPT-3 paper that established few-shot prompting (NeurIPS 2020) (https://doi.org/10.48550/arXiv.2005.14165)
Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (NeurIPS 2022) (https://doi.org/10.48550/arXiv.2201.11903)
Kojima et al., Large Language Models are Zero-Shot Reasoners: 'Let's think step by step' (NeurIPS 2022) (https://doi.org/10.48550/arXiv.2205.11916)
Claude docs: prompt engineering overview and techniques (https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview)
Claude docs glossary: temperature, sampling, and why temperature 0 is still not fully deterministic (https://platform.claude.com/docs/en/about-claude/glossary)
Liu et al., Lost in the Middle: How Language Models Use Long Contexts (TACL, vol. 12, 2024) (https://doi.org/10.1162/tacl_a_00638)
Laban, Hayashi, Zhou, and Neville (Salesforce AI Research and Microsoft), LLMs Get Lost in Multi-Turn Conversation: upfront vs. turn-by-turn specification (arXiv, May 2025) (https://doi.org/10.48550/arXiv.2505.06120)
Shi et al., Large Language Models Can Be Easily Distracted by Irrelevant Context: in-topic distractors hurt most (ICML 2023) (https://doi.org/10.48550/arXiv.2302.00093)
Claude Code commands reference: /clear starts a new conversation with empty context, /compact summarizes and continues the same one (https://code.claude.com/docs/en/commands)
3. Agentic IDEs and Vibecoding Workflows
Working inside Cursor and Claude Code on real codebases.
Chat in a browser doesn't scale to real projects. Agentic IDEs put the model inside your repository, where it can read, edit, and run code. This chapter builds working habits for agent mode, context, refactoring, large codebases, and deciding when to split work across parallel agents.
By the end of this chapter, you will be able to
Choose the right interaction mode (ask, plan, agent) for a given task
Supervise an agent mode session by checking each step's result before letting it continue, instead of leaving it unattended until it finishes
Control what a model can see in an IDE, and what it can't
Refactor in small verified steps, keeping restructuring separate from new features
Decide whether a task should be split across parallel agents or kept in one conversation you can steer
3.1 Cursor and Claude Code
Agentic editors differ from chat assistants in one structural way, which is that they can see and act on your actual files. They bring files into context, propose edits as diffs you can review before accepting, and run terminal commands directly, like installing a package, running the tests, or reading the error they just caused. Acting and then seeing the result of the action is what makes them feel different from a chat window.
Most of these tools offer some version of three modes, and knowing when to use each is a skill of its own. Ask mode answers questions about the code without changing it, which is the fastest way to understand a file or a bug before touching anything. Plan mode drafts an approach and waits for your approval. Agent mode, also called code mode, takes a goal and works through a sequence of edits and commands while you watch. Some tools add more (Cursor also has a Debug mode), but these three cover the decisions you'll make most often [1][18].
Codex is in the same category. OpenAI now ships it as the coding side of ChatGPT. The desktop and web apps can work on a project and drive a browser, and the separate Codex CLI and IDE extension bring the same agent into a terminal or an editor, with cloud sessions that run on OpenAI's servers [21]. Cursor and Claude Code take slightly different shapes. Cursor, a fork of VS Code (a copy of the editor's code, developed separately), starts from the editor and adds the agent. Claude Code starts from the agent and runs wherever you already work, in a terminal, an editor extension, a desktop app, or a browser. The mental model is the same in all of them, a capable collaborator with hands, working in your repo. The day to day differences are about which fits your existing workflow and how each exposes its controls.
Knowing when to use one mode over another mostly comes down to the size of the change. A typo fix doesn't need an agent planning across files, and a five file refactor shouldn't be done by hand one edit at a time. Agent mode on a tiny task wastes its overhead, and manual edits on a sprawling one waste yours.
The bottom rung of that ladder is typing the change yourself. Delegating even a one line fix still costs a prompt, a wait, and a check of the result, and on a change that is small and already fully formed in your head, that overhead exceeds the edit. Renaming one variable in one file, bumping a version number, or deleting a line you're looking at are keyboard work, and handing them to a model adds a review step to a task that didn't have one. Optional O returns to this calibration for readers who already code, where the reflex runs the other way.
Cursor's mode selector. Agent plans and executes. Plan drafts an approach first. Debug and Multitask are task-specific. Ask responds without changing files.
3.2 Agent mode in practice
In agent mode, the model behaves like a junior engineer working a ticket. It reads the relevant files, forms a plan, makes edits, runs commands to check its work, reads the output, and goes around again. Chapter 8 describes this same plan, act, observe loop for agents in general.
The power of that loop is also why it needs watching, and the watching is verification, not reading. Two moments matter most. Before the agent starts, read the plan it proposes (ask for one if the tool doesn't offer it) and fix any misreading of what you wanted. After it finishes, read its summary of what it did, then check the result yourself. Run the app, run the tests, and try the case you care about. Reading the generated code line by line is rare now, and this book won't pretend otherwise. The exception is code that handles money, permissions, or personal data, where a mistake hurts someone other than you. Have that reviewed, by a second model with fresh context, by a person, or both. Generated code needs the check more than it looks like it does. In Veracode's 2025 study of more than 100 models across 80 coding tasks, 45 percent of the generated code introduced a known security flaw, and the rate barely moved as models improved [17]. The failure to avoid isn't leaving the agent alone for a while. It's shipping whatever came back without checking that it does what you meant.
Agent mode goes wrong in two predictable ways, and both are yours to prevent. The first is a vague goal. 'Clean this up' gives the model no definition of done, so it picks one, and what it picks may not match what you had in mind. The second is a long run with no check-ins. Small misreadings compound, and by step fifteen the agent may be confidently building on a mistake from step two, or reaching for a command like 'rm -rf /' (which deletes every file on the machine) to make a stubborn error go away.
The fix for both is smaller steps. Even for a modest task, splitting the goal into pieces ('first migrate the data, then wire up the endpoint, then build the form') gives you a natural place to check each one before the next begins. A good agent run feels less like handing off a project and more like pair programming with someone fast who occasionally needs to hear 'wait, back up.'
Cursor in plan mode, pausing to ask clarifying questions before it writes any code. Catching ambiguity here is far cheaper than correcting a finished wrong run.
3.3 Context management
Context windows are finite, so an AI IDE is constantly choosing on your behalf what the model gets to see. That includes your open tabs, files you referenced explicitly, results from searching the project or the web, and, in tools that index your codebase, files pulled in because they look related. Knowing that this choice is happening is the first step to influencing it, because the model's answer depends on the part of your project it was actually shown.
There are two types of control. The first is inclusion. Most tools let you @-mention a file, or paste its path into the chat, to guarantee it's in context, so you can hand the model exactly the interface and example it needs instead of hoping it finds them. Paths also fix a failure that pasting causes. Chat inputs truncate long messages, often silently, so a pasted document can arrive with its ending cut off while everything that survived sits in the window whether the task needs it or not. A path hands the same material over losslessly, and an agentic tool can open it and read the parts that matter. When what you're about to paste runs longer than a screenful, save it to a file and give the model the path. The second is exclusion. Ignore rules, usually a dedicated file like Cursor's .cursorignore, keep generated bundles, lockfiles, and vendor library code out of the model's view entirely. The same lever reaches past the repository. Agentic tools can usually read any folder you point them at, so if an earlier project of yours, or an open source repo you've cloned, does something close to what you want, give the agent its path and say to do it the way that project does. A working example is a better spec than a paragraph, and the agent can read it directly instead of you pasting fragments. That reach is also why the permission boundary in 3.6 matters: an agent that can read your whole disk should be told which parts to touch.
Including too much hurts, and this is measurable. Models recall material near the start and end of a long input more reliably than material in the middle, and their accuracy drops as irrelevant text is added [14][15]. A 4,000 line generated file you don't care about can crowd out the 40 line file that holds the bug. The model can't read your mind about which of them matters, so feeding it what the task actually needs, and nothing else, is part of the job.
For larger projects, codebase indexing changes the picture. The tool embeds your whole repository so the model can pull in related files by meaning without you naming them (the same RAG idea Chapter 4 covers, applied to code). That is a real help on questions about unfamiliar code, and it still isn't a substitute for pointing at the right file when you know which one it is.
Both of those are choices you make per request. The third kind of context is persistent. Every major tool has a place for standing instructions that get injected into every request in a repository (Chapter 1 mentioned the names: Claude Code's memory, Cursor's project rules, Copilot's custom instructions) [11][12][20]. The question they all answer is what you would otherwise find yourself retyping at the start of every session. Your stack, the commands that build and test the project, and the directories that are off limits can be written down once. What doesn't belong is anything short lived. Notes about the bug you're chasing this afternoon would still be in the file long after it's fixed, spending context on every future session.
These instructions consume the same finite window as everything else, on every request, which is why this is a context topic and not a configuration one. Claude Code's documentation puts the target under 200 lines and notes that longer files reduce how reliably the instructions are followed [20]. Cursor, Copilot, and Claude Code all now let you scope rules to particular files or directories so they load only when relevant, which is worth using as soon as your instructions outgrow a page [11][12][20]. Rereading the file every few weeks and deleting what no longer applies keeps it honest, and each of those vendors gives the same advice.
3.4 Refactoring with AI
AI excels at mechanical refactors, such as the same change across a hundred files, a repeated block extracted into a function, comments reformatted in bulk, or deprecated libraries upgraded. These are the tedious transformations people are slow and sloppy at and models are fast and consistent at, provided you describe the change precisely and have a way to verify it, whether that's a test suite, a type checker, a clean build, or a screenshot of the working page.
AI is less reliable at deciding what should change in the first place. Whether a 900 line file quietly doing two jobs (checkout logic that now also formats emails) should become two files, or whether a complicated abstraction is worth keeping, depends on your plans for the codebase, which the model doesn't know unless you tell it. Left unspecified, it will pick a direction, so keep the what and the why on your side and delegate the how.
The safe pattern is to change structure and behavior in separate steps. Restructure first, with the code doing exactly what it did before, and confirm nothing broke, using the tests if the project has them and the running app if it doesn't. Only then add the new behavior. If one request both reorganizes the code and adds a feature, a failure could be either the reorganization or the unfinished feature, and you can't tell which. Keep them apart and every failure points at one cause.
3.5 Finding your way in large codebases
The hardest part of working in a big codebase is rarely making the change. It's finding where the change goes and what it might break. This orientation step is where AI assistants have changed the job most. Asking 'what is this module responsible for, and what calls it?' or 'trace a request from this endpoint to the database' turns an afternoon of reading unfamiliar files into a few minutes, and a controlled study of an assistant built into the IDE for exactly this purpose found that developers using it understood unfamiliar code better than those working from search and documentation alone [16].
Treat the answer as a map. The assistant can misread an unusual pattern or miss a side effect, so for anything important (the code that handles money, permissions, or data), ask it to list the code that calls the function, or have a second agent trace the same path on its own. If the two answers disagree, open the editor's references view and settle it yourself. That is still far faster than building the map by hand.
This applies to more than inherited code. More and more of what you'll orient inside is code an AI wrote a few prompts ago that nobody has read closely, so getting your bearings quickly in code you didn't write, whether a teammate's, an open source project's, or an agent's, is one of the most durable skills in this book. Optional Chapter E builds a unit around it.
3.6 Permissions, autonomy, and steering
An agentic editor can run commands, including ones like pushing to GitHub or deploying database rules, so every one of these tools puts a permission boundary between what the model wants to do and your machine [1][2]. By default they ask before consequential actions. You can widen or narrow that boundary so whole categories of action run with or without your approval, and where you set it depends on how reversible the actions are and how much you trust the tool on the task in front of you.
Every major tool also lets you switch the boundary off. Claude Code calls it bypass permissions mode, reached from the terminal with the flag --dangerously-skip-permissions or from a setting in its editor extension, and nicknamed 'YOLO mode' [2]. Codex calls it Full access (--yolo on the command line), and Cursor calls it Run Everything [18][19]. The names are honest about the trade. With approvals off, the agent can delete files, push commits, spend money against an API, or follow a malicious instruction hidden in a file it reads, without pausing for you. That is why Chapter 14's material on prompt injection matters here. The less you're in the loop, the more damage one hijacked instruction can do.
Turning approvals off also gets a lot of work done, which is why so many developers run that way daily. Clicking approve on every command is slow, and an agent that can run its own tests and builds without waiting for you can carry a task to the end while you work on something else. The decision is a trade, not a rule. On a personal project, a prototype, or a repository with nothing you can't restore from git, the speed is usually worth it. When the session can touch production credentials, payment code, user data, or files you can't get back, keep the prompts on for that session, or narrow the allowlist instead of dropping it. Decide it per task, on purpose, rather than leaving one setting on forever because you forgot it was there.
The agent's reach extends past the repository to the machine itself. A useful first run in a new environment is to ask it to set your computer up for vibecoding: install the runtimes and command line tools you're missing, configure git, and check that the editor, its extensions, and the shell agree with each other. It saves an afternoon, and it's also the clearest example of why the boundary exists, because the agent is now installing software and editing configuration outside any project, where nothing is restorable from git. Keep approvals on for that run and read what it proposes to install before you say yes.
Supervising an agent is more than approving or rejecting its finished work. While a run is in progress, you can course correct. Interrupt the moment it heads somewhere wrong instead of letting it finish a plan you already know is doomed, and use the queue most tools offer to add a follow up instruction that takes effect after the current step. Interrupt early. A wrong turn caught at step three is fixed with one sentence, while the same wrong turn caught at step thirteen has usually spread through the session and is cheaper to restart than to unwind.
Two habits cost nothing and save a lot. First, write long or complex prompts somewhere other than the chat box, or copy them before sending, so that if the run goes sideways or the tool glitches and loses your message, you can start a fresh session and send the same prompt again, or a tweaked one. Second, ask the model to ask you clarifying questions before it starts on anything large. A single 'what should happen if the user reloads mid checkout?' up front prevents a whole implementation built on the wrong assumption.
There's also a way to step out of the prompt by prompt loop entirely. Claude Code, Codex, and Cursor all have a '/goal' command that gives the agent a finish line instead of a single task [13][22]. You state a completion condition, and the agent keeps working, checking, and revising until the condition is met. You might write '/goal the signup form validates email and password on the server, every test in the auth module passes, and the linter reports nothing.' Claude Code doesn't let the agent grade its own work here. A separate, cheaper model reads the session at intervals and decides whether the goal has been reached [13]. That evaluator only sees what appears in the session and runs nothing itself, so the condition has to be something the session can show, such as tests passing or a clean build, and not 'make good, no mistake.' That last one is a fine prompt for the caveman skill (a real plugin at caveman.so, whose pitch is 'why many token when few do trick') and a useless goal, because nothing in the session can show it was met. Given a checkable goal, the agent can carry a large task to completion on its own. Given a vague one, it will either stop early or convince itself it succeeded.
The 'Allow Dangerously Skip Permissions' toggle, found in the Claude Code extension's settings panel in VS Code or Cursor. It's the same switch as the --dangerously-skip-permissions flag in the terminal.Codex asks the same question with three levels. Ask for approval checks with you before editing outside the project or using the internet, Approve for me only asks about actions it flags as risky, and Full access is the equivalent of bypass permissions.
Approvals off is a per task decisionTurning approvals off (Claude Code's --dangerously-skip-permissions, Codex's Full access, Cursor's Run Everything) lets the agent delete files, push commits, spend money, or follow a hidden malicious instruction without pausing. It's fine, and fast, on a prototype or a repository you can restore. Turn it back on for any session that can reach production credentials, payments, or user data.
3.7 Running several agents at once
Every tool in this chapter can now run more than one agent at a time, whether that means several sessions side by side, a lead agent handing pieces of work to helpers, or a fleet the tool spawns from a single instruction. The appeal is obvious. Four agents working while you're away from the keyboard or steering a fifth sounds like four times the output, and sometimes it is. The rest of the time it's four times the changes with none of the watching, and the question to settle before you split anything is which of those you're about to get.
The cost is the supervision from 3.6. You can watch one conversation as it works, but not four, so parallel agents can take wrong turns, or irreversible actions, that nobody catches in real time. A lead agent that reports back on what its helpers did softens this, and it still only tells you afterward. So take care when you spawn several agents. The overhead, the token spend, and the chance of an uncaught mistake all scale with the count.
Parallel agents work well when each one has a piece of the job that doesn't overlap with the others. Bugs in separate parts of the project, distinct parts of one large feature, reading several codebases to answer a question, or the same task run several ways to compare approaches or models all qualify, because no agent's work can undo another's. Read only work is the safest of all. Anthropic's write up of the multi agent system behind Claude's research feature draws the boundary in the same place. Such systems do well on tasks that split cleanly, on information too large for one context window, and when many tools are in play, and they do badly where every agent needs the same context or the agents depend on each other. Most coding tasks, they add, have fewer genuinely parallel pieces than research does [4].
Anthropic also published a demonstration of that boundary. One of its researchers pointed sixteen agents at writing a C compiler from scratch. While the remaining work was a pile of separate failing tests, splitting it was easy, since each agent could take a different test. When the remaining work became compiling the Linux kernel, one enormous task that doesn't divide, the sixteen agents stopped helping. Every one of them hit the same bug, fixed the same bug, and overwrote each other's changes [8].
So keep the work on one agent you can watch when the steps depend on each other, when two agents would touch the same files, or when a mistake would be expensive to undo. Authentication, payments, and anything involving user data belong in one watched conversation. A sensitive change does deserve more eyes, and the eyes should be yours.
When you do run agents in parallel, no two of them should be pointed at the same files at the same time, and version control makes the rest safe, because each agent's branch records its changes and you merge them afterward. The usual way to get that isolation is a git worktree, a second checkout of the same repository with its own files and branch [5]. The tools increasingly set this up for you (Claude Code creates a worktree for a background agent on its own, and cloud agents run in isolated copies by definition [5]). GitHub is plain about what happens without it in its own fleet feature. Sub agents share a filesystem with no file locking, so if two write the same file, the last to finish silently wins [9].
Two different settings get confused here. Reasoning effort gives one agent more thinking time on a hard problem (Optional M covers the progression). Parallel agents give you more hands on work you already understand and simply have a lot of. Claude Code's ultracode setting turns on both at once, a high effort level plus automatic splitting of the work across agents [6]. A hard problem you can't yet specify calls for effort. Forty files needing the same change call for parallel agents. Use parallel agents on the first kind and you get several confidently wrong answers instead of one.
Then there's the bill. Anthropic measured its research system at about fifteen times the tokens of an ordinary chat, where a single agent doing the same job used about four times, so the multi agent part alone roughly quadrupled the cost [4], and Claude Code's documentation warns that a team of agents costs several times a single session because each agent carries its own context window [7]. Optional Chapter Q covers the cost side in more detail. None of this makes parallel agents wrong. It means the speed has a price, and the price should be a decision you make rather than a toggle you didn't know was on.
It's also worth remembering that most of what's published about how well this works comes from the companies selling it, which makes the evidence against it unusually credible. One peer reviewed study held the prompts, tools, and token budget constant and varied only how the agents were coordinated, across 260 combinations of task suite, coordination structure, and model (six task suites, five ways of organizing the agents, and three model families, where a family is one vendor's line of models). On SWE-bench Verified, a standard suite of real GitHub issues used to test coding agents, every multi agent arrangement scored below a single agent of the same model, and the broader finding was that the stronger the single agent, the less coordination adds [10]. Try it on your own projects and draw your own conclusion.
The question to ask before you split the workAsk yourself whether you would accept each agent's work, for whatever task you're on, without having watched it happen. If the honest answer is no, the task wants one agent and your full attention, not four agents you'll audit afterward.
Summary
Agentic IDEs put the model next to your code, where it can read files, propose diffs, and run commands. We covered picking a mode to fit the size of the change, supervising an agent while it works, controlling what the model can see, refactoring in small verified steps, and getting oriented in code you didn't write. We ended with running several agents at once, which gets more done on work whose pieces don't overlap, at the price of watching any one of them closely. The habit that runs through all of it is deciding on purpose: what the agent may do without asking, when to interrupt, and when a task deserves your full attention.
Key terms
Agent mode. A mode, also called code mode, where you send a prompt stating a goal and the model plans, edits files, and runs commands across multiple steps, typically while you supervise.
Multi-file edit. A single request that touches several files at once, such as swapping a logo on every page or changing a function signature plus every call site, the kind of change agent mode exists for.
Diff. A representation of exactly which lines a change adds, removes, or modifies, and what you review before accepting an AI edit.
Codebase indexing. The tool's automatic building of a searchable representation of a whole project (often via embeddings) so the model can find relevant files without being told where they are.
Context window management. Deciding which files and references the model holds in working memory at once, since that capacity is finite.
Ignore rules. Configuration (such as Cursor's .cursorignore file) that excludes files from the model's context so irrelevant or huge files don't crowd out what matters.
Overarching instructions. Project files that give an AI tool standing instructions, such as the command that runs the tests, the naming conventions to follow, or the directories to leave alone, injected into every request in that repository.
Refactoring. Restructuring code without changing what it does (renaming, extracting functions, reorganizing), verified before any new behavior is added.
Permission boundary. The approval layer an agentic IDE puts between the model's proposed actions (running commands, editing files) and your machine, which you can widen or narrow.
Goal (/goal). A command that sets a completion condition and lets the agent keep working until the condition is met, shifting execution from prompt by prompt to condition based. The goal is judged from the session's own output, so it must be something that output can show, like every test in a module passing or a build with no errors.
Bypass permissions mode. A setting that lets an agent act without asking for approval (Claude Code's --dangerously-skip-permissions flag, nicknamed 'YOLO mode'; Codex's Full access; Cursor's Run Everything). Fast on a prototype, risky in any session that can reach real credentials or data.
Steering. Interrupting an agent mid-run to correct its course, without waiting for it to finish a plan you can already tell is wrong.
Parallel agents. Splitting one job across several agents working at the same time. You get more done on pieces that don't touch each other, and you give up the ability to watch any one of them closely.
Git worktree. A second checkout of the same repository, with its own files and its own branch, so two agents can edit at once without overwriting each other.
Check your understanding
Q1. You need the same one line fix applied in forty unrelated files, and, separately, you need to track down one bug that only appears when two requests arrive at the same time. Which is the better candidate for parallel agents?
The timing bug, because a hard problem benefits most from several agents attacking it at once
Both equally, since parallel agents shorten any task that would otherwise run long
Neither, because parallel agents are only appropriate for read-only work such as search
The forty files, because the pieces don't touch each other and no agent's work undoes another's
Answer: D. Parallel agents pay off on independent pieces like the forty files. The timing bug is one problem whose steps depend on each other and needs steering, so it belongs in one conversation you can watch. Read-only work is the safest thing to run in parallel, but it isn't the only thing.
Q2. What structurally distinguishes an AI-native IDE from a browser chat assistant?
It can read and act directly on your filesystem, including running commands
It runs on a fundamentally different and smarter class of model
It works entirely offline with no network connection required
There's no structural difference, only a difference in branding
Answer: A. The defining capability is direct read/write access to your project plus command execution, not a different underlying model.
Q3. When does agent mode most commonly go off the rails?
When it's given a precise, narrow goal and told exactly what to change
When the goal is vague or it runs a long time with no check-ins
When context is curated well, and the right files are in scope
Agent mode is reliable enough that it rarely goes off track
Answer: B. Vague goals and long unsupervised runs both let small mistakes pile up before anyone notices.
Q4. What's the safest order of operations when refactoring with AI assistance?
Refactor and add the new feature in one request to save a round trip
Skip verification, since a refactor doesn't change behavior
Refactor alone, verify, then add behavior as a separate step
Rewrite the whole file from scratch so the result stays consistent
Answer: C. Keeping the refactor in its own step means any failure points at the restructuring and nothing else.
Q5. When is turning approvals off (bypass permissions, 'YOLO' mode) a reasonable choice?
Always, since approval prompts only slow you down
Whenever the task involves production credentials, payments, or user data
Never, since an agent should not run anything without a human approving it
On a prototype or a repository you can restore, for a task you'd otherwise be clicking approve on all afternoon
Answer: D. With approvals off, the agent can delete files, push commits, spend money, or follow a hidden malicious instruction without pausing. That's an acceptable trade for speed on disposable or restorable work, and a bad one for any session that can reach real credentials or data.
Q6. What makes a '/goal' completion condition usable rather than counterproductive?
Something the session's output can show, like every test in the auth module passing or a build with no errors
It should stay broad and open-ended so the agent can use its judgment
It should avoid naming tests or builds, which constrain the agent too early
Any phrasing works about equally well, since the agent infers your intent
Answer: A. Because 'done' is judged from what shows up in the session, a fuzzy goal like 'make it good' lets a run stop early or declare success without much to show for it. A checkable condition gives that judgment something real to work from.
Practice
Exercise 1. Open a codebase you didn't write (a teammate's repo or an open source project). Ask an AI assistant to explain what one function is responsible for and what calls it. Then, for once, do it the manual way too, with project wide search (Ctrl+Shift+F in most editors) and 'find references,' and compare the two. (Hint: Write down what each pass told you about the function, and note where they differed. The comparison is the point. The assistant is faster, and the references view is what proves which code actually calls the function, including any callers the assistant missed.)
Exercise 2. Give an agent-mode tool a deliberately vague goal ('make this better') on a small file, then a precise one ('extract the validation logic into its own function'). Compare what each run actually did. (Hint: Look specifically at how much the vague run changed that you didn't ultimately want.)
Exercise 3. Start an agent-mode run, then interrupt it the moment its plan looks wrong and redirect it, instead of letting it finish. Compare the cost of correcting early against what it would have taken to unwind a completed wrong run. (Hint: The point is to feel the difference in cost between correcting early and unwinding a finished wrong run.)
Exercise 4. Pick a task with two clearly independent parts. Run it once as a single agent told to do both and not to spawn helpers, then once with parallel agents allowed. Time both, and count how much of each result you actually verified before accepting it. (Hint: The number worth writing down is the second one. Parallel agents usually win on the clock and lose on how much of the result you personally checked.)
Sources
Claude Code documentation: install, settings, and permissions (https://code.claude.com/docs/en/overview)
Claude Code documentation: permission modes, including bypassPermissions and the --dangerously-skip-permissions flag (https://code.claude.com/docs/en/permission-modes)
Model Context Protocol (MCP), the open standard for connecting AI tools to external data and tools (https://modelcontextprotocol.io)
Anthropic engineering, How we built our multi-agent research system: where multi-agent helps, where shared context makes it a poor fit, and the token multiple it costs (https://www.anthropic.com/engineering/multi-agent-research-system)
Claude Code documentation: git worktrees for running agents in parallel without conflicting edits (https://code.claude.com/docs/en/worktrees)
Claude Code documentation: dynamic workflows, and the ultracode setting that pairs high effort with workflow orchestration (https://code.claude.com/docs/en/workflows)
Claude Code documentation: what parallel agents cost, including the token multiple for a team of agents (https://code.claude.com/docs/en/costs)
Carlini, Building a C compiler with a team of parallel Claudes: where sixteen agents parallelized well and where they collided (Anthropic engineering, February 2026) (https://www.anthropic.com/engineering/building-c-compiler)
GitHub, Run multiple agents at once with fleet in Copilot CLI: sub-agents share a filesystem with no file locking, so the last writer wins silently (https://github.blog/ai-and-ml/github-copilot/run-multiple-agents-at-once-with-fleet-in-copilot-cli/)
Kim et al., Capable language models can outgrow the benefits of collaboration: a compute-matched comparison of single-agent and multi-agent architectures across 260 configurations, including SWE-bench Verified (Nature Machine Intelligence, July 2026) (https://doi.org/10.1038/s42256-026-01268-y)
Cursor documentation: project rules in .cursor/rules, including glob scoping and rules invoked by name (https://cursor.com/docs/context/rules)
GitHub documentation: repository custom instructions for Copilot, including applyTo path scoping (https://docs.github.com/en/copilot/how-tos/configure-custom-instructions/add-repository-instructions)
Claude Code documentation: the /goal command, its completion conditions, and the separate evaluator model that judges them (https://code.claude.com/docs/en/goal)
Liu et al., Lost in the Middle: How Language Models Use Long Contexts (TACL, vol. 12, 2024) (https://doi.org/10.1162/tacl_a_00638)
Nam et al., Using an LLM to Help With Code Understanding: a controlled study of an in-IDE assistant for understanding unfamiliar code (ICSE 2024) (https://doi.org/10.1145/3597503.3639187)
Veracode, 2025 GenAI Code Security Report: 45 percent of AI-generated code samples across 100+ models and 80 tasks introduced a known security flaw (July 2025) (https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/)
Cursor documentation: run modes, including Auto-review, Allowlist, and Run Everything (https://cursor.com/docs/agent/security/run-modes)
OpenAI, Agent approvals and security: Codex's approval modes, from read-only to full access, and the --yolo flag (https://learn.chatgpt.com/docs/agent-approvals-security)
Claude Code documentation: how Claude remembers your project, with the under-200-line target for CLAUDE.md and path-scoped rules in .claude/rules (https://code.claude.com/docs/en/memory)
OpenAI, ChatGPT and Codex documentation: the desktop and web apps, Codex CLI, IDE extension, and cloud sessions (https://learn.chatgpt.com/docs)
Cursor changelog, August 19, 2026: the /goal command, which gives the agent a long-lived objective to work toward until it is complete (https://cursor.com/changelog/08-19-26)
4. Giving AI Context
Context windows, RAG, tool calling, MCP, and knowledge bases.
A model can only draw on two things: what it learned during training, which is frozen, and what is in front of it right now, which you control. This chapter covers every way to put more in front of it. The context window is the space that holds it. Retrieval augmented generation (RAG) fetches the relevant parts of your own documents into that space. Tool calling lets the model ask for a function to be run. The Model Context Protocol (MCP) is the standard for plugging those tools in. A knowledge base is the curated collection of documents that RAG and those tools draw from.
By the end of this chapter, you will be able to
Explain what a context window and a token are, and why both are finite resources
Describe how a RAG pipeline retrieves and grounds answers in your own data
Explain how tool calling lets a model act through functions you define
Identify when MCP and a knowledge base would improve an AI application
4.1 Context windows and tokens
Everything a model knows in the moment comes from two places, its training (frozen at a cutoff date) and its context window. The context window is everything the model can consider at once, meaning your prompt, any files or images you attached, the conversation so far, and its own reply as it writes it [2]. Nothing outside that window exists to the model, which is why so much of working with one comes down to deciding what goes in it.
The window is measured in tokens rather than words or characters. A token is the unit a model actually reads, usually a few characters or a piece of a word, so 'cat' might be one token and 'caterpillar' three. Both your context limit and your API bill are counted in tokens, which explains a lot of otherwise confusing behavior and cost.
Because the window has limited capacity, something has to be cut when it's nearly full. Some tools drop the oldest messages, others summarize them, and either way, details from early in a long session quietly stop being available to the model, including instructions you gave twenty messages ago.
The size a vendor advertises for its window is a ceiling on how much the model will accept, and not a guarantee of how much it can reason over. Windows are now big enough that the problem looks solved, but benchmarks that ask a model to connect facts scattered through a long input, instead of finding one planted sentence, show many models losing accuracy well before their claimed length [6].
Position matters as well. Across a long input (the whole prompt as the model sees it, conversation included), material near the start and the end is recalled more reliably than material in the middle [7], so when a prompt is long, put the instruction that actually drives the answer near the end.
The strategies that address all of this come up through the rest of the book. Split long documents into pieces, called chunks, and give the model only the relevant ones. Carry a summary of an old conversation into a new one instead of the whole transcript. Put in the handful of files the task actually needs and leave out the ones included 'just in case.' A bigger window is not automatically a better answer. A window stuffed with marginally relevant text makes it harder for the model to find and use the part that matters, a failure that has come to be called context rot [5].
A related habit is the sidechat, a branched conversation started from a highlighted piece of a response. Most chat tools now support it. ChatGPT calls it branching in a new chat, and Google AI Studio calls it branch from here [17]. The branch inherits the history up to that point, so a quick definition, a tangent, or a one-off subtask gets answered with full context while the main thread stays clean. That matters because everything in the main thread occupies its context window and steers later answers, so detours accumulate as noise. Research on chat interfaces makes the same case, that a single linear thread makes tangents costly, and branching lets you explore them without losing the main line of work [18]. When a detour turns out to matter, paste its conclusion back into the main chat, not the whole exchange.
What fills a context window. Every one of these competes for the same finite space, along with the reply the model is writing, and the model can only work from what's in it.
4.2 Retrieval-Augmented Generation (RAG)
A model can't know about anything outside its training data, which includes your company's internal documents, this textbook, and last week's logs. Putting all of it in the context window rarely fits, and it would bury the relevant part in noise even when it did. Retrieval augmented generation (RAG) is the standard fix. It finds the parts of your documents that relate to the question and gives the model just those, so the answer is grounded in your material instead of the model's memory [1].
RAG has two stages. The first happens ahead of time, before anyone asks a question. Your documents are split into chunks, each chunk is turned into an embedding (a list of numbers that captures its meaning, so passages about similar things get similar numbers), and the embeddings are stored in a database built for finding the nearest ones. The second stage happens each time a question arrives. The question is turned into an embedding the same way, the database returns the chunks whose embeddings are closest to it, and those chunks are added to the prompt ahead of the question, so the model answers from specific passages rather than from vague recollection. This is semantic search, search by meaning instead of by exact characters (the way Ctrl+F or a regular expression matches). A question about 'refund policy' can retrieve a passage titled 'returns and reimbursements' even though the two share no words.
Embeddings have a blind spot, though. Strings that carry no meaning, like error codes, function names, product codes such as SKUs (stock keeping units, the identifiers stores use for inventory), and surnames, are exactly what plain keyword search finds best, because it matches the characters and never has to understand them. A good pipeline runs both kinds of search, merges the results, and then applies a reranker, a second model that scores each candidate passage against the question and keeps the best matches [16]. On one published benchmark, adding keyword search alongside embeddings cut the share of questions whose answer never appeared in the top twenty results from 5.7 percent to 2.9 percent, and adding a reranker took it to 1.9 percent [8]. If you build retrieval that anyone depends on, you'll end up using both.
The ways RAG fails are predictable. Chunks that are too big bury the relevant sentence in noise, and chunks that are too small lose the context needed to make sense of them. If retrieval brings back the wrong chunks, the model answers confidently from them anyway, because it has no way to know that better evidence exists, the same blindness to its own errors that Chapter 9 examines. So RAG reduces hallucinations without eliminating them, and a well built system cites the passages it retrieved so a human can check them, which Chapter 9 will insist on.
Chunk โ Embed โ Retrieve โ Answer
A RAG pipeline. Ahead of time, documents are chunked and embedded. When a question arrives, it's embedded the same way, the closest chunks are retrieved, and the model answers from them.
Check whether you need RAG at allMeasure your documents before you build anything. If the whole collection fits comfortably in the window, put all of it in the prompt and skip the pipeline. Anthropic's guidance draws that line at roughly 200,000 tokens, about 500 pages, and prompt caching (Optional Q) keeps that cheap, since the same block of documents at the start of every prompt is charged at a discount after the first time it's sent [8]. Below that line, RAG adds an indexing job, an embeddings bill, and a new way for the answer to be wrong.
4.3 Tool calling
RAG gives a model more to know. Tool calling gives it a way to act, by letting it ask for a specific, predefined function to be run (looking up an order in a database, for example) and receive the result before it answers.
The mechanics are more constrained than they first sound. The model never executes anything itself. It emits a structured request [3] that says, in effect, 'call the function named getWeather with the input {city: "Tokyo"}.' Your code, or the platform, decides whether to honor that request, runs the function, and returns the result to the model. That separation is a security boundary, because you choose which tools exist and what each is allowed to do, so the model can only act through doors you deliberately opened.
You're the one who defines those doors. A tool has three parts, a name, a plain English description of what it does, and a schema listing the inputs it takes and their types [3]. The model reads the description the way it reads a prompt, which is why most tool calling failures live there. If two tools have descriptions that sound alike (say, search_docs and find_documents), the model will sometimes pick the wrong one, and the fix is to merge them into one tool or to give related tools a shared prefix, like calendar_create and calendar_list, so their roles are obvious. That works better than adding yet another tool [9]. Two details carry forward to Chapters 7 and 8. Whatever a tool returns comes back through your code, so you can check or filter it before the model sees it. And most platforms now offer a strict setting that guarantees the inputs the model produces match your schema exactly, which removes the checking and retrying that used to surround every call [10].
Tool calling also turns a model's weaknesses into someone else's job. Models make arithmetic mistakes and know nothing after their training cutoff, but give one a code execution tool and a web search tool and it can hand exactly those tasks to systems that do them well.
Agents (Chapter 8) are this mechanism in a loop, a model calling tools repeatedly until a goal is met, and the AI features you'll build in Chapter 7 are tool calls wired to real APIs. Understand tool calling now and those chapters will feel like assembling parts you already know.
4.4 Retrieval as a tool call
Once retrieval is itself a tool, the decision about what to fetch moves from you to the model. Instead of embedding an entire collection of documents in advance, an agent keeps a set of pointers (file paths, URLs, saved searches) and loads a document only when it decides it needs one [11]. You've already seen this in Chapter 3. A coding agent that searches and reads its way through a repository is doing retrieval through tool calls, with no embeddings built ahead of time.
Two things get better this way. Nothing has to be indexed in advance or kept up to date, so the agent always sees the current files, and the agent can search again after noticing that its first result was wrong, which a fixed pipeline can't. Two things get worse, speed and cost, since every search is another trip through the model, and over a large collection a prebuilt index answers far more cheaply per question. The published comparisons find that giving a model the whole document tends to win on quality and lose badly on cost [12], which is why choosing between the two per question beats committing to either up front.
4.5 MCP and knowledge bases
Tool calling creates a practical problem of its own. Every application has to write its own code to connect each model to each data source (one connector for your calendar, another for your database, another for Slack), and none of it can be reused, because each connector was written for one app and one model's way of calling tools. The Model Context Protocol (MCP) is an open standard that fixes this by defining one common interface between models and external tools and data, so a connector written once works in any app that speaks the protocol [4].
MCP began at Anthropic, and in December 2025 Anthropic handed it over to the Agentic AI Foundation, a fund under the Linux Foundation co founded with Block and OpenAI and backed by AWS, Google, Microsoft, and others, so that no single company controls it [13][14]. That matters for a practical reason. An interface one company could change or withdraw at will would not be worth building on.
Two things are worth knowing about an MCP server before you install one. The first is what it offers. A server can expose tools the model can call, resources it can read, and prompts, which are reusable templates the application can hand to the model [15]. A server that offers only resources is still useful, since it supplies context without the model running anything. The second is where it runs. A local server runs on your own machine and talks to the application through its standard input and output, while a remote server runs on someone else's computer, talks over HTTP, and authorizes you with OAuth. Installing a remote connector therefore hands a third party a scoped credential to a real account, a security decision whose consequences Chapter 14 covers. A running joke in the ecosystem is that the S in MCP stands for security.
RAG pipelines and MCP connected tools both draw on a knowledge base, the curated collection of documents the model can search, and its quality sets a ceiling on the answers. A knowledge base that's split into sensible chunks, kept up to date, and free of contradictions produces grounded, trustworthy answers, while a stale or messy one produces confident answers built on bad evidence, which is worse than no answer at all.
The main idea of this chapter is one shift in how to think about a model. Stop treating it as a fixed answer machine you query, and start treating it as something you connect to the right context and the right tools. Context windows, RAG, tool calling, and MCP are four ways of doing that, and a model's usefulness in practice is set by how smart it is and, just as much, by what you connect it to.
Summary
A model can only use what its training taught it and what you show it in the moment. We covered the context window and its limits, then retrieval augmented generation (RAG), which splits your documents into pieces, finds the pieces relevant to a question, and hands them to the model before it answers. We also covered tool calling, the Model Context Protocol (MCP) for plugging tools in, and why a clean knowledge base matters. RAG reduces hallucinations without eliminating them, so a good system cites its sources for you to check.
Key terms
Token. The unit a model reads and writes, usually a few characters or a piece of a word. Vendors count both context limits and API prices in tokens, so the same word covers what fits in the window and what you pay for.
Tokenization. Splitting text into tokens before a model processes it.
Context window. The maximum number of tokens a model can consider at once.
Context rot. Degradation in output quality when a context window is filled with too much marginally relevant content.
RAG (Retrieval-Augmented Generation). Retrieving the parts of your own documents that relate to a question and giving them to the model before it answers, so the answer is grounded in that material instead of the model's memory.
Embedding. A numeric vector that captures the meaning of a piece of text so that similar meanings have nearby vectors.
Vector database. A store optimized for finding the embeddings closest to a query embedding, the retrieval engine behind RAG.
Semantic search. Searching by meaning, via embeddings, so a query matches passages that say the same thing in different words.
Chunking. Splitting documents into pieces of a suitable size before embedding, so retrieval returns focused, relevant passages.
Tool calling / function calling. A mechanism where the model requests that a defined function be run with specific inputs and receives the result back into the conversation.
MCP (Model Context Protocol). An open standard for connecting models to external tools and data sources so one integration can be reused across many AI apps.
Grounding. Tying a model's answer to verifiable external data (retrieved docs, tool results) instead of relying on its trained memory.
Knowledge base. The curated, searchable collection of documents behind RAG and many connected tools.
Check your understanding
Q1. What happens once a conversation exceeds the model's context window?
The model automatically remembers everything regardless of length
Older or less relevant content gets cut or summarized to make room
The API call always fails with no way to continue
Context windows are unlimited for all models
Answer: B. The window is a finite token budget, so something has to give, usually the oldest or least relevant content.
Q2. In a RAG pipeline, what's retrieved when a question arrives?
The model's full training dataset
A random sample of all stored documents
Document chunks whose embeddings are closest to the question's embedding
Nothing, since RAG retrieves only after the answer is generated
Answer: C. Retrieval is a similarity search over embeddings. The chunks closest in meaning to the question get pulled in before the model answers.
Q3. What does tool calling actually let a model do?
Directly execute arbitrary code on your machine with no oversight
Permanently modify its own training weights
Bypass the context window entirely
Request that a specific, defined function be run and receive its result
Answer: D. The model requests a named tool call with inputs. Your code or platform actually executes it and returns the result.
Q4. What problem does MCP (Model Context Protocol) solve?
It standardizes the model-to-tool interface, so one integration is reusable
It speeds up generation by compressing the data a model receives
It removes the need for a context window by streaming data on demand
It's a competitor to RAG that removes the need for embeddings entirely
Answer: A. Without a shared standard, every app needs its own connector code per data source. MCP defines a common interface so one server can be reused, like a USB port for AI tools.
Q5. Why can a stale or messy knowledge base be worse than having no knowledge base at all?
It can't be worse, because more retrievable data is always better
It produces confident answers grounded in outdated evidence, which misleads
A stale knowledge base causes the retrieval step to fail with an error
A knowledge base has no measurable effect on the quality of answers
Answer: B. A knowledge base that's outdated or contradictory still gets cited with confidence, so the answer looks trustworthy while resting on bad evidence.
Practice
Exercise 1. Build a tiny RAG demo by hand. Take three short text files and split each into a few chunks. Then write three different questions and, for each, work out which chunk would be retrieved and whether the answer is actually in it (reason about similarity by hand for this exercise). (Hint: Include at least one question whose answer lives in only one file, and one whose wording shares no words with the passage that answers it. Check that the 'closest' chunk is the right one both times.)
Exercise 2. Describe one task in your own life where tool calling (a model plus a function it can call) would beat a model with no tools at all. (Hint: Look for a task that needs live, current data the model couldn't have memorized, like today's weather or your account balance.)
Sources
Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, the original RAG paper (NeurIPS 2020) (https://doi.org/10.48550/arXiv.2005.11401)
Claude docs: context windows, token accumulation, and context rot (https://platform.claude.com/docs/en/build-with-claude/context-windows)
Tool use with Claude: how models request tool calls and receive results (https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview)
Model Context Protocol, the open standard for connecting AI applications to external tools and data (https://modelcontextprotocol.io)
Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models? (COLM 2024) (https://doi.org/10.48550/arXiv.2404.06654)
Liu et al., Lost in the Middle: How Language Models Use Long Contexts (TACL, vol. 12, 2024) (https://doi.org/10.1162/tacl_a_00638)
Anthropic, Introducing Contextual Retrieval: the 200,000-token threshold for putting a whole knowledge base in the prompt, and top-20 retrieval failure rates for embeddings, embeddings plus BM25, and reranked hybrid retrieval (September 2024) (https://www.anthropic.com/news/contextual-retrieval)
Anthropic engineering, Writing effective tools for agents: descriptions as prompts, namespacing, and why overlapping tools distract a model (https://www.anthropic.com/engineering/writing-tools-for-agents)
Claude docs: structured outputs, strict tool use, and schema conformance through constrained decoding (https://platform.claude.com/docs/en/build-with-claude/structured-outputs)
Anthropic engineering, Effective context engineering for AI agents: just-in-time context loading, lightweight identifiers, and the attention budget (https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)
Li et al., Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (EMNLP 2024, industry track) (https://doi.org/10.48550/arXiv.2407.16833)
Linux Foundation press release: formation of the Agentic AI Foundation, anchored by MCP, goose, and AGENTS.md, with its founding and platinum members (December 9, 2025) (https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation)
Anthropic, Donating the Model Context Protocol and establishing the Agentic AI Foundation (December 2025) (https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation)
Model Context Protocol architecture: hosts, clients, and servers, the tools, resources, and prompts primitives, and stdio versus streamable HTTP transports (2026-07-28 revision) (https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture)
Nogueira and Cho, Passage Re-ranking with BERT: the reranking step that scores retrieved passages against the query (arXiv, 2019) (https://doi.org/10.48550/arXiv.1901.04085)
Suh, Min, Palani, and Xia, Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models (UIST 2023) (https://doi.org/10.1145/3586183.3606756)
5. Frontend Vibecoding
Generating real UIs with React, design systems, v0, and Lovable.
The frontend is where AI generation feels like magic and where 'looks right' can hide broken behavior. This chapter pairs React fundamentals with AI UI generation and human-centered design. The goal: students ship interfaces that are both pretty and usable.
By the end of this chapter, you will be able to
Read and explain the props, state, and composition in a generated React component
Identify structural inconsistency in AI-generated UI and fix it with design tokens
Apply human-centered design and accessibility basics to an AI-generated interface
Distinguish a polished demo from a usable, accessible product
5.1 React fundamentals
React organizes UI around a few ideas you need to recognize even when you're not typing them [1]. Components are reusable, self-contained pieces of interface, such as a button, a card, or a whole page. Props are the read-only inputs you pass into a component to configure it, like a label or a color. State is data a component owns that changes over time, and the key behavior is that updating state triggers a re-render so the screen reflects the new value automatically. You describe what the UI should look like for a given state, and React keeps the pixels in sync.
Modern React layers a few more concepts on top. Hooks are functions like useState and useEffect that let a component hold state and run side effects, such as syncing with a browser API or setting up a subscription [20]. Fetching data is the side effect to be careful with, because React's own guidance points you at your framework's data loading or a caching library instead of a hand-written effect [20]. Composition is the practice of building a complex screen out of many small components, never one giant one. It's the main thing that keeps a growing app readable, and it's why generated code is usually split into lots of little files.
Here's why this matters in a book about not typing code by hand. You don't need to author this from scratch anymore, but you need to be able to read it when something breaks. When a generated form mysteriously doesn't update, the bug is almost always a state or props misunderstanding, and you can't diagnose what you can't recognize. The floor is the fluent-reader bar, lower than the fluent-author bar but not zero.
Here's a concrete tell: an AI-generated component you can't explain (what its state is, why each piece of state exists, what its props are) is a component you can't safely modify or debug. Throughout this chapter, the recurring move is to generate fast, verify that it works, and read closely enough that you could explain what each piece of state and each prop is for, because that is what you'll need the day it breaks.
A UI as a tree of components. Naming this structure up front gives the model a map to build against.
From a rough sketch to a working interface, the path most frontend sessions follow.
5.2 UI generation with v0 and Lovable
Tools like v0 and Lovable turn a description or even a rough sketch into working, styled components or entire screens [2][3]. This is the part of vibecoding that feels most like magic, because the feedback is instant and visual. You type 'a pricing page with three tiers,' and seconds later you're looking at one. Even centering a div, the task a whole generation of developers joked they couldn't do, now takes one polite sentence. For getting from a blank page to something clickable, nothing is faster.
But there's a characteristic failure to watch for: generated UI is usually visually polished and structurally inconsistent, with one button styled in inline CSS and the next with utility classes, and the spacing between cards is 16px here and 20px there, and two screens that should feel identical use subtly different shades. Each screen looks fine in isolation, which is exactly why the inconsistency is easy to miss until you see them side by side.
The reason is that the model generates each request fresh, optimizing for 'does this look right on its own,' never for 'does this match the other forty screens.' It has no enforced memory of the decisions it made an hour ago, which isn't a flaw you can prompt your way out of entirely. It's a structural property of generating UI piecemeal, which is exactly why the next section on design systems matters.
Use these tools for what they're great at: exploring options quickly and producing a first draft to react to, treating the output as clay rather than a finished sculpture. The value is in how fast it gets you to something concrete, not in shipping it untouched.
5.3 Design systems and tokens
A design system is the shared vocabulary a product's UI is built from: a fixed color palette, a spacing scale (say, multiples of 4px), typography rules, and a library of reusable components everyone draws from. Its whole job is consistency, making a hundred screens feel like one product designed by one team, rather than a hundred demos.
Design tokens are the mechanism that makes a design system enforceable. A token is a named variable for a design decision (color-primary, spacing-4, radius-md) referenced everywhere instead of hardcoded values. The payoff is twofold: changing the brand color becomes editing one definition instead of hunting through forty files, and a component that uses tokens literally can't drift out of step with the rest of the app, because it's pulling from the same source of truth.
This is the direct antidote to the inconsistency problem from the last section. After you generate a first draft, the real work is refactoring it onto your tokens and component library, replacing the model's one-off colors and ad-hoc spacing with references to the system. It's less glamorous than generating the screen, but it's what converts ten impressive demos into one coherent product.
AI can actually help with this step too, if you point it at the system: 'rewrite this component to use our existing Button and our spacing tokens' is a well-bounded refactor (Chapter 3) that models do well. The judgment that stays yours is defining the system in the first place (deciding the palette, the scale, the component set), because that's a product identity decision, not a mechanical transformation.
Pointing a model at your system has stopped meaning describing it in a prompt. The current practice is to make the system retrievable. A component registry, public or private and company-internal, can be exposed over MCP so an agent browses and installs from your own library when it needs a Button [5]. A design tool can expose its variables and its component-to-code mappings the same way, so what comes back carries your token names where a screenshot would have produced raw hex codes [6]. A generator can be taught the system once so that every later generation builds against it [7]. One rule turns this from plumbing into a guarantee. A component, prop, or token that cannot be verified from those sources should not be used. That is Chapter 4's grounding, pointed at your UI.
The token file is what makes this portable. The design token format reached its first stable version in October 2025, with theming and multi-brand sets built in, so a palette and a spacing scale now travel as an interchange format instead of a per-tool convention [8]. That is what turns handing your system to a model into a real move, though better plumbing still doesn't decide the palette.
5.4 Responsive layout and themes
A generated screen is built for the one viewport the prompt implied, almost always a wide desktop one, because that is the state that training data over-represents and the only state anyone is going to look at. Responsive design, building an interface that holds up from a phone to a desktop, is the first thing that quietly doesn't happen. The demo runs on a laptop, and the first real user opens the app on a phone.
A media query asks how wide the window is, while a component's real constraint is how wide its slot is. A card that reads well in a main column breaks when the same component is dropped into a sidebar, even though nothing about the window changed. Available across browsers since February 2023, a container query lets a component adapt to whatever container it lands in, which is what component-driven generation needs [9]. Ask for it by name when a component will be reused in more than one place.
Themes are that same token layer cashed in. Inverting the colors is not a theme. A dark theme is a second full set of values for the same token names, which is why a codebase with a token layer can gain one in an afternoon, and a codebase of hardcoded hex codes can't. The stable token format carries both sets in one file [8]. In CSS, you declare which schemes a page supports with color-scheme [10], then pick the per-token value for each with light-dark(), one declaration carrying both, available across browsers since May 2024 [11].
5.5 Showing the model what it made
A model writes CSS and never renders it. So when it reports that a layout should look right, that is a prediction about pixels it hasn't seen. Nothing in the generation loop closes that gap unless you close it, which matters more here than almost anywhere else in the book, because on the frontend, appearance is the requirement. (The old joke goes: two CSS properties walk into a bar, and a barstool in a completely different bar falls over. The model has read that joke thousands of times. It has still never seen the bar.)
There are two ways to close it. You can be the eyes, pasting the render back the way Chapter 10 has you paste a screenshot of a bug, the difference being that there you know something is wrong, and here you are finding out. Or you can give the agent a browser it can drive, at which point the check becomes something it performs instead of something you remember to do: resize to a phone width, switch the emulated color scheme to dark, screenshot each, run a Lighthouse audit for accessibility and performance [12], all against the states the last section said nobody checks. Which agents can drive a browser is Section 11.6's question.
A visual loop tightens the match without guaranteeing it. The peer-reviewed benchmark for turning a reference screenshot into working code finds that models lag hardest at recalling elements from the reference and at generating correct layouts [13], the two failures that a quick glance won't reliably catch either. 'The agent looked at it' is a better bar than 'the agent claimed it,' though neither replaces you looking.
5.6 Human-centered design and accessibility
A demo only has to work once, for one audience, under ideal conditions: the presenter's laptop, fast wifi, a happy path clicked in a rehearsed order. A product has to work for many different people in conditions you don't control: someone on a slow connection, on a cracked phone in bright sunlight, using a screen reader, or doing the one thing in the form you didn't expect. Human-centered design is designing for that real range instead of the demo's fantasy.
Three fundamentals carry most of the weight. Visual hierarchy: size, weight, and placement should make it obvious what matters most, so a user's eye lands in the right place without effort. Then feedback: every action needs a visible response. A button that does nothing on click is a bug even when the underlying request actually succeeded, because the user has no way to know. And forgiveness: clear error states and the ability to undo turn mistakes from dead ends into recoverable moments.
Accessibility (often written a11y) is the part most likely to get skipped, and the part with real consequences. We have watched student demos that looked gorgeous on a projector and fell apart the moment someone tabbed through them. The basics are concrete and learnable [4]: sufficient color contrast so text is readable, full keyboard navigation so the app works without a mouse, meaningful alt text on images, and semantic markup so assistive technology can make sense of the page. They're what make your product usable by people with disabilities, and they overlap heavily with what makes software hold up for everyone.
The scale is measurable and moving the wrong way. An annual scan of the top million home pages found detectable WCAG failures on 95.9 percent of them in February 2026, up from 94.8 percent a year earlier and averaging 56.1 errors per page, which reversed years of gradual improvement [14]. Low-contrast text led by a distance, on 83.9 percent of pages, and the report names heavier reliance on third-party frameworks and AI-assisted coding, vibe coding in its own words, among the likely causes. The same scan kills the obvious fix of asking for more ARIA. ARIA attributes per page rose 27 percent in a year, and pages using ARIA averaged more errors than pages without it, so the request is about as likely to add wrong ARIA as to help.
Prompting for accessibility up front does buy something. A controlled comparison of accessibility-agnostic against accessibility-oriented prompts found measurable gains, notably in focus visibility and in the markup of information and relationships, with barriers persisting in semantic structure, which is the part that cannot be patched on afterward [15]. The bigger lever sits with vendors, since aligning a model for accessibility during training cuts its inaccessibility rate by about 60 percent [17]. So ask, then measure, starting with the rule you can check in seconds, 4.5:1 contrast for normal text at Level AA [16].
AI-generated UI reliably nails the first impression and skips accessibility, for a telling reason: contrast ratios and keyboard support don't show up in the screenshot the model is implicitly optimizing toward. So treat an accessibility pass as a required, named step after generation (tab through the whole flow with no mouse, check contrast, confirm every image has alt text), not as optional polish you'll get to later, because later is where accessibility goes to die.
Accessibility became law before it became habitFor a whole class of products, the accessibility pass stopped being a quality bar someone could waive under deadline. Requirements under the European accessibility directive have applied since June 28, 2025 and reach e-commerce and consumer banking services, including businesses outside the EU selling to EU consumers, with conformance assessed against the harmonized European standard that points at WCAG [18][19]. Where those requirements apply, skipping the accessibility pass is no longer a choice a shipping product gets to make.
Summary
AI generates UI fast. The real skill is governing that output. We covered React fundamentals and learned to tame inconsistency with design tokens. We also treated accessibility as a requirement of the first draft instead of later polish. Generated screens look fine alone but drift apart in a set, so the work is consolidating them onto shared patterns and testing the things a glance misses: keyboard navigation.
Key terms
Container query. A CSS rule that styles a component by the width of the container it sits in instead of the width of the browser window, so the same component adapts wherever it is placed.
Color scheme. The set of light and dark presentations a page declares support for, with a value chosen per token for each, so one token layer carries both themes.
Component. A reusable, self-contained piece of UI in React. Complex interfaces are built by composing many small components.
Props. Read-only inputs passed into a component to configure what it renders.
State. Data a component owns that can change over time. Updating it triggers a re-render so the UI reflects the new value.
Hooks. React functions (like useState, useEffect) that let components manage state and side effects.
Composition. Building larger UI by combining small components, the main way React code stays readable as an app grows.
Design system. The shared set of colors, spacing, typography, and reusable components a product's UI is built from.
Design tokens. Named variables (e.g. color-primary, spacing-4) that encode design decisions so one change updates every usage.
Responsive design. Building UI that adapts to different screen sizes, from phone to desktop.
Accessibility (a11y). Designing so screen reader users, keyboard users, and people with low vision can all use the product, including contrast, keyboard nav, alt text, and semantic markup.
Component library. A reusable collection of pre-built UI components that enforces consistency across screens.
v0. Vercel's AI builder that turns a text description or sketch into working, styled React components and into complete full-stack apps it can deploy for you.
Lovable. An AI tool that generates entire working screens or apps from a natural-language description, fast enough to feel like magic.
Check your understanding
Q1. Why is it risky to ship an AI-generated component you can't explain?
It isn't risky, generated code is always correct
AI-generated components never compile
You can't safely modify or debug something you don't understand
Explaining components is only needed for job interviews
Answer: C. Reading and understanding generated code is what lets you safely change it later or diagnose a bug when one appears.
Q2. What problem do design tokens solve?
They reduce bundle size, so the application loads noticeably faster
They replace the need for a reusable component library entirely
They apply only to backend code and have no role in the interface
They centralize design decisions so one change updates every usage at once
Answer: D. Tokens are named variables for design values. Change the token definition once, and every component using it updates consistently.
Q3. Why does AI-generated UI often skip accessibility basics?
Accessibility problems don't show up in a screenshot, so they're easy to miss
Accessibility markup is technically impossible for a model to generate
Accessibility matters far less than the visual design of the page
Models aren't capable of writing useful alt text for an image
Answer: A. A screenshot looks polished regardless of contrast ratios or keyboard support, so accessibility needs a deliberate, separate check.
Q4. Why does AI-generated UI tend to be visually polished but structurally inconsistent across screens?
These tools randomize styling deliberately so the result looks creative
The model generates each request fresh, with no memory of earlier styling choices
Inconsistency appears only when the prompts themselves are poorly written
React itself introduces the inconsistency through its rendering model
Answer: B. Each generation optimizes for looking right on its own, not for matching forty other screens, so nothing keeps color, spacing, or component choices consistent across sessions.
Q5. What's the recommended way to use tools like v0 and Lovable?
Ship what they generate straight to production without further changes
Avoid them entirely and hand-code every component you need yourself
Use them to explore options fast, then treat the output as a draft to refine
Use them only for backend scaffolding and never for interface work
Answer: C. These tools are fastest at getting from a blank page to something concrete. The value is in speed to a first draft, not in the polish of the unedited output.
Practice
Exercise 1. Generate a small UI (e.g. a settings form) with v0 or Lovable, then refactor it to use a consistent spacing scale and check it with a keyboard only (no mouse). (Hint: Tab through every interactive element. If you can't tell what's focused, that's an accessibility gap to fix.)
Exercise 2. Take a generated component and write, in your own words, what its props and state are and why each piece of state exists. (Hint: If you can't explain why a piece of state exists, that's worth asking the model, or removing.)
v0 by Vercel: AI-powered UI and app generation (https://v0.app)
Lovable documentation: building apps from natural-language descriptions (https://docs.lovable.dev/introduction/welcome)
W3C WAI: Accessibility Principles (contrast, keyboard access, text alternatives) (https://www.w3.org/WAI/fundamentals/accessibility-principles/)
shadcn/ui documentation: an MCP server for browsing, searching, and installing from public and private component registries (https://ui.shadcn.com/docs/mcp)
Figma: the Dev Mode MCP server, exposing components, variables, and Code Connect mappings to coding agents (https://www.figma.com/blog/introducing-figma-mcp-server/)
v0 documentation: Design Systems 2.0, teaching a generator your real components and tokens (https://v0.app/docs/design-systems-2)
Design Tokens Community Group: the Design Tokens specification reaches its first stable version (2025.10), with theming and multi-brand support (https://www.w3.org/community/design-tokens/2025/10/28/design-tokens-specification-reaches-first-stable-version/)
MDN: the @container at-rule, styling a component by its container instead of the viewport (https://developer.mozilla.org/en-US/docs/Web/CSS/@container)
MDN: the color-scheme property, declaring which color schemes an element supports (https://developer.mozilla.org/en-US/docs/Web/CSS/color-scheme)
MDN: the light-dark() color function, one declaration carrying both theme values (https://developer.mozilla.org/en-US/docs/Web/CSS/color_value/light-dark)
Chrome DevTools MCP tool reference: emulate (color scheme and viewport), resize_page, take_screenshot, and lighthouse_audit (https://github.com/ChromeDevTools/chrome-devtools-mcp/blob/main/docs/tool-reference.md)
The WebAIM Million, February 2026: an accessibility analysis of the top one million home pages (https://webaim.org/projects/million/)
When LLM-Generated Code Perpetuates User Interface Accessibility Barriers, How Can We Break the Cycle? (Web for All 2025) (https://doi.org/10.1145/3744257.3744266)
W3C WCAG 2.2, success criterion 1.4.3 Contrast (Minimum): 4.5:1 for normal text at Level AA (https://www.w3.org/WAI/WCAG22/quickref/#contrast-minimum)
A11yn: aligning language models for accessible web UI code generation (https://doi.org/10.48550/arXiv.2510.13914)
Directive (EU) 2019/882, the European Accessibility Act: accessibility requirements applying from 28 June 2025 (https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32019L0882)
European Commission: the European Accessibility Act, scope of covered products and services (https://commission.europa.eu/strategy-and-policy/policies/justice-and-fundamental-rights/disability/union-equality-strategy-rights-persons-disabilities-2021-2030/european-accessibility-act_en)
React documentation: the useEffect reference, including why data fetching belongs outside an effect (https://react.dev/reference/react/useEffect)
6. Backends, Databases, and Authentication
Supabase, Firebase, data modeling, auth, and CRUD apps.
A real app needs to store data and know who its users are. This chapter takes students from a static frontend to a full-stack CRUD application: backed by a database, with working authentication.
By the end of this chapter, you will be able to
Choose between a relational and a document-store backend for a given data shape
Model entities and relationships before writing schema or backend code
Decompose a feature into CRUD operations and the access rules each one needs
Distinguish authentication from authorization and test for authorization gaps
6.1 Supabase vs. Firebase
A frontend is a beautiful, forgetful thing: refresh the page, and everything is gone. To become a product, an app needs a backend, somewhere to store data permanently and somewhere to decide who's allowed to see it. Supabase and Firebase are the two most common batteries-included answers, meaning they bundle a database, authentication, and usually file storage into one platform you can stand up in minutes.
They differ in how they store data. Supabase is built on Postgres, a relational database [1]: your data lives in tables with defined columns, types, and relationships, and you query it with SQL (or a generated client that writes the SQL for you). Firebase's Firestore is a document store [3]: data lives in flexible, JSON-like documents grouped into collections, with no enforced schema up front. That flexibility is liberating early and can become a liability later, when you wish something had been enforcing structure all along. Firebase has since softened that split by shipping a managed Postgres option alongside Firestore [16], so choosing the platform no longer decides the data model for you. The relational versus document question is still the one to answer, and answering it deliberately beats inheriting it from whichever platform you signed up for.
Neither is universally better. They fit different shapes of data. Relational shines when your data has clear structure and relationships (orders belonging to users, comments belonging to posts) because the database itself can enforce and efficiently query those connections. Document stores shine when data is naturally nested, variable, or evolving fast, and when you'd rather not commit to a rigid schema while you're still figuring out the product.
Because both are batteries-included, an AI assistant can scaffold a working backend from a plain description in a single session, with tables, auth, and basic queries wired up before lunch. That speed is worth using, and it also raises the stakes on the rest of this chapter: when the backend appears this easily, it's tempting to not look closely at the schema it chose or the access rules it did (or didn't) write, which is exactly where the expensive mistakes hide.
Measurement has caught up with that warning. A June 2026 audit of 200 deployed applications built by AI coding agents found at least one vulnerability in 90 percent of them, and broken access control was the single largest class, at 36 percent of every finding, with 82.8 percent of those access-control failures sitting in backend code rather than the frontend [5]. A benchmark of 392 backend-generation tasks arrived at the same place from the other direction, successfully running exploits against roughly half of the backends that were functionally correct, which means code that passed its own tests was still exploitable [6]. Put the two together, and the practical lesson is that the backend is where the failures of generated code concentrate, so 'it works' is measurably not evidence that it's safe.
The life of a request, from a click in the browser to a row in the database and back.
The layers of a typical web app. Vibecoding touches all of them, so it helps to know which layer a bug lives in.
6.2 Data modeling
Data modeling is the decision of what your data actually looks like: what entities exist (users, posts, orders), what fields each one has, and how they relate (one user has many posts, one order belongs to exactly one user). It's the closest thing a typical app has to architecture, and it deserves a few minutes of real thought before any code gets generated.
The reason to get it right early is asymmetric cost. Changing a data model on a whiteboard is free. Changing it after the app has real users and real data is a migration. You have to move existing data into the new shape without losing or corrupting any of it, often while the app is live. A model that was 'good enough' under deadline pressure can tax every future feature, because everything you build sits on top of it.
A simple, durable habit prevents most of the pain: write the model in plain English first. List every entity, then state every relationship as a sentence: 'a post belongs to a user,' 'a post has many comments,' 'a user can like many posts.' Those sentences translate almost mechanically into tables and foreign keys (relational) or collection structure (document), and writing them out surfaces ambiguities (can a comment belong to another comment?) while they're still cheap to resolve.
This is also a place to direct the AI rather than defer to it. If you hand the model your entity-and-relationship list, it can generate a sound schema quickly. If you ask it to 'design a database for a blog' with no guidance, it'll invent a model that may not match the product in your head, so the thinking stays yours while the typing is the model's.
Direction covers the shape of the data and skips the rules. A generated schema is reliably permissive about those: columns left nullable that the product cannot work without, no unique constraint where a duplicate would be a bug, and foreign keys with no delete behavior stated. Four things are worth naming in your own words before you accept a schema: NOT NULL on every field the feature genuinely requires, UNIQUE wherever two of a thing is a bug, CHECK on any value with a real range, and an explicit delete rule on every relationship [7]. The delete rule matters most, since Postgres defaults to refusing the delete and raising an error, while CASCADE silently removes every child row. Neither behavior is safe to inherit by accident, since the right answer depends on whether a child row means anything once its parent is gone. These rules belong in the database and not in application code. The database is the one gate that every path passes through, including the admin panel, the migration script, and whatever the model writes next week, so a constraint is the only rule that survives code you haven't read yet.
6.3 Migrations
A migration is the script that carries a live database from one schema to the next, adding a column, changing a type, or splitting a table in two. Application code runs the same way every time you deploy it, while a migration runs once, against data that already exists. Rolling a bad deployment back, which Chapter 13 covers, restores your code. No rollback restores rows that a migration deleted, so this is the rare change that has no undo.
The trap is mechanical. A tool generating a migration from your declared schema cannot tell a renamed column from a deleted one plus a new one, so the default output for 'rename this field' is a DROP and an ADD that takes the column's contents with it [8]. Postgres stacks on a second hazard, since changing a column's type normally rewrites the whole table and its indexes under an ACCESS EXCLUSIVE lock, which turns a data-loss risk into a downtime risk on any table of real size [9].
This is where 'it works on my machine' means the least in the whole book. The standard local workflow rebuilds the database from the migration files against an empty schema [10], so every migration passes locally for the simple reason that there was nothing there to lose. Nothing in the output distinguishes that clean local rebuild from the same command pointed at production, where it would be a catastrophe.
Three habits cover most of the exposure. Read the generated SQL yourself and look for the four verbs that destroy things: DROP, ALTER COLUMN TYPE, TRUNCATE, and any DELETE without a WHERE clause. Run the migration against a copy carrying real-shaped data before it touches the real database. And keep a backup that you have actually restored at least once, because a backup that nobody has restored is a belief about your data instead of a safeguard for it.
6.4 CRUD operations
CRUD (Create, Read, Update, Delete) names the four operations that cover nearly everything an app does to its data. It's a small idea with outsized usefulness, because once you see it, most features visibly decompose into it: 'post a comment' is a Create, 'see your order history' is a Read, 'edit your profile' is an Update, 'remove a bookmark' is a Delete.
Naming a feature in CRUD terms early tells you exactly what backend you need to build. A feature is really just some specific combination of these operations against specific entities, plus rules about who can do which. 'A user can manage their own bookmarks' unpacks into create/read/update/delete on the bookmark entity, scoped to the current user, which is concrete enough to start building.
CRUD also gives you a checklist for completeness and for security. For each entity, ask which of the four operations should exist and who's allowed to perform each, and notice that the verbs carry different risk, since reads leak data, deletes destroy it, and updates can quietly corrupt it. Walking the CRUD grid per entity is a fast way to catch the operation you forgot to protect, which sets up the authorization discussion next.
6.5 Authentication and authorization
These two words sound alike and mean different things, and conflating them is one of the most common and most dangerous mistakes in app building. Authentication answers 'who is this user': sign-up, sign-in, and maintaining a session (typically a token, like a JWT, that proves you're still logged in on later requests so you're not re-entering a password on every click). Modern apps often delegate this via OAuth ('Sign in with Google') so they never handle raw passwords at all.
Authorization answers a completely different question: 'is this authenticated user allowed to do this specific thing?' Being logged in doesn't mean being allowed to read every other user's private messages, delete someone else's post, or view the admin dashboard, because authentication only gets you in the front door, while authorization decides which rooms you can enter once inside.
Here's the error pattern to burn into memory, because AI-generated auth code walks straight into it: models reliably get authentication working and quietly skip authorization. You'll get a polished login flow, feel done, and ship an app where, underneath, any logged-in user can query any other user's data, because the row-level security rules that scope data per user were never written [2][4], so the app looks secure when it isn't.
The reason those rules carry so much weight here is architectural. In a batteries-included backend, the browser talks to the database directly, holding a key that is published on purpose, so no server sits in the middle quietly filtering rows. Those rules are the entire perimeter, which Firebase's documentation states outright, calling Security Rules the only safeguard blocking access for malicious users [11]. Turning the rules on and writing them are separate acts, so a table with rules enabled and no policy denies everything, which reads as a broken app and often gets 'fixed' by switching them off, while a table where the rules were never enabled is readable by anyone holding the published key. Enabling isn't uniform across platforms either, and on Supabase, a table created through the dashboard has row-level security on by default, while a table created in raw SQL does not [2], which is precisely the path an AI writing migrations takes. Alongside the publishable key sits a secret key that bypasses every policy by design [12], so code that resolved a permissions error by reaching for it removed the perimeter instead of fixing the permissions. The consequences are documented in CVE-2025-48757, scored 9.3 critical, for row-level security insufficient enough to let unauthenticated attackers read and write arbitrary tables in one AI builder's generated sites [13].
So treat authorization as something you verify by attacking, not something you assume the generated code handled. The concrete test: log in as User A and deliberately try to read or modify User B's data through your own API. If it works, you have a hole, and you'd rather find it than have a stranger find it. This 'break your own access rules' habit returns in Chapter 14, because it's one of the highest-value security checks a builder can run.
What a working login still leaves openFour checks on an app whose login already works, none longer than five minutes. Is email confirmation still switched off, which speeds up a demo and lets anyone register under an address they don't control [15]? Does any query trust a user id read out of the request body instead of the verified session, which is broken access control, still the number one entry on the OWASP Top 10 [14]? Does the deployed build authenticate with the publishable key or the bypass-everything one? Was the table created in SQL, where the rules may never have been switched on? Each of these leaves the login screen working perfectly, which is why none of them show up in a demo.
Summary
The backend is where data lives and where mistakes get expensive. We modeled data in plain language first, then implemented CRUD. We drew the line between authentication (who you are) and authorization (what you may touch). The lesson that matters most: verify access rules like an attacker, since an AI's first-draft security rules often let one user reach another's data.
Key terms
Supabase. A batteries-included backend platform built on Postgres, bundling a relational database, authentication, and file storage that you can stand up in minutes.
Firebase. A batteries-included backend platform whose Firestore database is a document store, bundling data storage, authentication, and file storage.
CRUD. Create, Read, Update, and Delete, the four basic operations on data that most app features decompose into.
Data model. The decision of what your data looks like: what entities exist, what fields each has, and how they relate, ideally written in plain English before any schema or code.
Schema. The defined structure of your data: which tables/collections exist, what fields they have, and how they relate.
Relational vs. document. The two dominant ways to store data: relational databases enforce tables with defined columns and relationships (Supabase/Postgres), while document stores hold flexible, schema-less JSON-like documents (Firebase/Firestore).
Relational database. A database (e.g. Postgres, used by Supabase) where data lives in tables with defined columns and relationships, queried with SQL.
Document store. A database (e.g. Firestore, used by Firebase) where data lives in flexible, nested documents grouped into collections.
Authentication. Verifying who a user is, through sign-up, sign-in, and proving identity on later requests.
Authorization. Deciding what an authenticated user is allowed to do or see, distinct from authentication and a common source of security holes.
Session. State that keeps a user logged in across requests, typically via a token, so they don't re-enter a password every action.
JWT (JSON Web Token). A signed token commonly used to carry a user's identity and claims between client and server.
OAuth. A standard protocol for letting users log in via a third party (e.g. 'Sign in with Google') without sharing their password.
Row-level security (RLS). Database rules that restrict which rows a given user can read or write, the mechanism that stops one user from seeing another's private data.
Check your understanding
Q1. What's the core structural difference between Supabase and Firebase?
Supabase provides authentication but no database of its own
They're effectively identical products sold under different names
Firebase supports mobile applications only and not the web
Supabase is relational (Postgres), Firebase is a document store (Firestore)
Answer: D. Supabase gives you structured tables and SQL. Firebase's Firestore gives you flexible, loosely structured documents in collections, and that contrast is still what the two platforms are best known for, though Firebase now sells a managed Postgres option as well.
Q2. Which CRUD operation does 'view your order history' correspond to?
Read
Create
Update
Delete
Answer: A. Viewing existing data without changing it is a Read operation.
Q3. What's the key difference between authentication and authorization?
They're two names for the same underlying security check
Authentication is who the user is, and authorization is what they may do
Authorization is always resolved before authentication runs
Only authentication genuinely matters for application security
Answer: B. Being logged in (authenticated) does not automatically grant permission to every action or every piece of data (authorization). Both must be enforced.
Q4. Why should you write a data model in plain English before generating any schema or code?
Plain English descriptions are a required input for Supabase and Firebase
AI models are unable to read an existing database schema directly
It surfaces ambiguities while they're still cheap to fix, before a migration
It offers no real benefit and is mostly a documentation formality
Answer: C. Changing a data model on a whiteboard is free; changing it after real users have real data requires a migration. Writing relationships as sentences first catches ambiguity early.
Q5. What's the most common way AI-generated backend code fails on the security front?
It usually fails to compile against the database client
It leaves both sign-up and sign-in unimplemented entirely
It can't establish a connection to the database at all
It gets sign-in working but skips the rules that scope data per user
Answer: D. Models reliably produce a working login flow but often skip the access rules that scope data per user, leaving an app that looks secure but isn't. You have to verify authorization by testing it like an attacker.
Practice
Exercise 1. Sketch a data model (entities + relationships, in plain English) for a simple bookmarking app, then build it with CRUD operations in Supabase or Firebase. (Hint: Write the relationships as full sentences first ('a bookmark belongs to a user') before creating a single table or collection.)
Exercise 2. After building sign-up/sign-in, deliberately try to access another (fake) user's private data through your own app's API while logged in as a different user. Does it work? Fix it if so. (Hint: This is testing authorization, not authentication. You're already logged in, and the question is whether you're allowed to see someone else's data.)
Sources
Supabase docs: every project is a full Postgres database (https://supabase.com/docs/guides/database/overview)
Firebase docs: Cloud Firestore, a NoSQL document database (https://firebase.google.com/docs/firestore)
Firebase docs: Security Rules for protecting data access (https://firebase.google.com/docs/rules)
Understanding the (In)Security of Vibe-Coded Applications: an audit of 200 deployed AI-agent-built applications (https://doi.org/10.48550/arXiv.2606.23130)
BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025), 392 backend-generation tasks with end-to-end exploits (https://doi.org/10.48550/arXiv.2502.11844)
PostgreSQL docs: constraints, including NOT NULL, UNIQUE, CHECK, and foreign-key ON DELETE behavior (https://www.postgresql.org/docs/current/ddl-constraints.html)
Prisma docs: customizing migrations, and why renaming a field generates a DROP plus an ADD (https://www.prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations)
PostgreSQL docs: ALTER TABLE, table rewrites and the ACCESS EXCLUSIVE lock (https://www.postgresql.org/docs/current/sql-altertable.html)
Supabase docs: database migrations and the local-to-production workflow (https://supabase.com/docs/guides/deployment/database-migrations)
Firebase docs: Security Rules basics, rules as the only safeguard on direct client access (https://firebase.google.com/docs/rules/basics)
Supabase docs: understanding API keys, publishable versus secret (https://supabase.com/docs/guides/getting-started/api-keys)
CVE-2025-48757: insufficient row-level security allowing unauthenticated read and write of generated sites' tables (https://nvd.nist.gov/vuln/detail/CVE-2025-48757)
OWASP Top 10:2025, A01 Broken Access Control (https://owasp.org/Top10/2025/)
Supabase docs: production checklist, including email confirmations and row-level security on all tables (https://supabase.com/docs/guides/deployment/going-into-prod)
Firebase docs: SQL Connect, a managed Cloud SQL for Postgres database for Firebase apps, formerly Data Connect (https://firebase.google.com/docs/sql-connect)
7. APIs and External Integrations
REST, AI APIs, webhooks, third-party services, and payments.
Most software value comes from connecting things. This chapter teaches students to consume and combine APIs, including AI APIs and payment providers, into apps that do something useful.
By the end of this chapter, you will be able to
Read API documentation for authentication, rate limits, and error handling before integrating
Call an AI API from a backend securely, keeping API keys server-side
Recognize the ways AI coding tools default to hardcoding or leaking keys, and know to rotate a key the moment it's exposed
Build an idempotent, signature-verified webhook handler
Explain why payment amounts must always be verified server-side, never trusted from the client
Compare mainstream payment processors and identify when a merchant-of-record provider is the right choice
7.1 REST APIs and third-party services
Almost no useful app is an island. The interesting value usually comes from connecting to services someone else already built (maps, email, payments, weather, AI), and the most common way to do that is a REST API, which exposes operations over HTTP organized around resources. The shape is predictable: GET /users/42 reads a user, POST /users creates one, the verb says what kind of operation and the path says on what, and once you've seen a few, most feel familiar.
Here's the counterintuitive part for an AI-assisted course: reading the documentation matters more now, not less. The model can write a syntactically perfect request to any API instantly, but it can't know the things that are specific to that API and often absent or outdated in its training, such as the exact authentication scheme, the current rate limit, the precise shape of an error response, and the one required header everybody forgets, which is why the docs are the source of truth and the model is just fast fingers.
Three details bite beginners repeatedly. Authentication: most APIs require an API key (or token) sent in a header, and they reject you politely if it's missing or malformed. Rate limits: services cap how many requests you can make per minute or per day, and exceeding the cap returns an error instead of your data. Error handling: a sturdy integration handles the documented failure shapes explicitly, because the network and the remote service will fail sometimes, full stop.
The classic avoidable bug lives in the gap between 'works once' and 'works in production.' A request that succeeds the first ten times can fail on the eleventh because you hit a rate limit, or the service had a blip, or a field you assumed was always present came back null, and generated code tends to cover the happy path and skip these. Adding the error and retry handling is exactly the kind of verification work this book keeps asking you to own.
Your app โ Request + key โ External API โ Handle / error
A call to a third-party API. The key authenticates you, and the error path is the part beginners forget to handle.
Most apps are glue between services. Each integration is another API your app has to call and handle.
7.2 Calling AI APIs
Calling a model API (Claude, OpenAI, and others) from your own backend is, structurally, just another REST call: you send a list of messages and get a response back, and that familiarity is the point of framing it this way. Everything you already know about authentication, error handling, and rate limits applies directly, so the AI feature you're adding is less exotic than it sounds, being an API integration where the API happens to be a language model.
Two AI-specific capabilities are worth using deliberately. Streaming returns the response token-by-token as it's generated, which is what powers the 'typing' effect and makes a slow answer feel responsive instead of frozen. And tool calling (Chapter 4) works here exactly as described: you define functions your backend can run, the model requests them when useful, and you feed the results back. This is how you give an AI feature access to your own data and actions rather than just its trained knowledge.
Now the rule that matters most, stated plainly: provider API keys live on the server, never in the browser, for concrete reasons. Anything shipped to the client, even 'hidden' in JavaScript, is visible to anyone who opens dev tools, and AI usage is metered and billed. An exposed key means a stranger can run up your bill. The fix is to route AI calls through your own backend endpoint that holds the key.
Finally, AI calls are billed per token, so cost is a design concern, not an afterthought. A feature that sends a giant prompt on every keystroke, or pipes a 50-page document into the model for a one-line answer, can be functionally fine and financially ruinous at scale, which Chapter 13 returns to. For now, internalize that every call has a price tag and design accordingly.
This is a spot where vibecoding actively makes the mistake more likely. Ask an AI tool to 'add the OpenAI API call,' and the fastest path to a working demo is often to paste the key straight into the source file, because that's the shortest route to code that runs. The model isn't being careless on purpose, since it's optimizing for 'this compiles and returns a response,' and a hardcoded string does that. You have to be the one who insists on the safer route (an environment variable, read server-side) even when the shortcut would demo just as well.
Once a key is hardcoded, it escapes through more doors than beginners expect. Chapter 13 covers what to do once one is committed [5]. Pushing to a public GitHub repo means bots are actively scanning for exposed keys, and studies that planted credentials as bait saw them picked up and used within minutes [1][2]. Committing a key remains the fastest known way to discover that your repository was public all along. This isn't a rare event: GitGuardian counted 28.6 million new secrets leaked on public GitHub during 2025, a 34 percent jump over the year before, and credentials for AI services grew faster than any other category, up 81 percent [3]. And it doesn't have to be committed to leak: pasting a file into an AI chat, a Slack message, or a screenshot for a bug report (Chapter 10's own advice) can just as easily hand a key to whoever reads it, if you didn't notice it was sitting in the code you shared. A key pasted into an AI chat also has a second life you don't control, because the conversation is stored on the provider's servers, and if that provider is ever breached, your key is part of what leaks. Many tools warn you when they spot one in your input. The safe habit is to treat any key that has passed through a chat as exposed, and rotate it once things are confirmed working.
If a key ever leaks, rotate itEnvironment variables and .gitignore prevent leaks; they don't undo one. If a key was ever committed, pasted into a chat, or shown in a screenshot, treat it as compromised: revoke and regenerate it in the provider's dashboard immediately, even if you 'caught it fast' or deleted the message. A key that touched any tool outside your own head should never be trusted again.
7.3 Webhooks
Most API interaction is you asking a service for something. Webhooks flip that around: the service calls you when something happens on its end. You register a URL, and the provider sends an HTTP request to it whenever an event occurs, like a payment succeeding, a file finishing processing, or a subscription renewing. It's push instead of pull, and it's how your app finds out about things that happen outside it without constantly asking.
The alternative is polling. You ask 'anything new yet?' on a timer. Contrasting the two clarifies why webhooks exist. Polling is simpler to reason about but wasteful and laggy: poll too often and you burn requests on 'nope,' poll too rarely and you react late, whereas webhooks deliver the news the instant it happens with no wasted calls, at the cost of a public endpoint and a bit more care to secure it.
That care starts with trust. Your webhook URL will eventually be discoverable, and anyone who finds it can send it fake events, a forged 'payment succeeded,' for instance. So providers cryptographically sign each payload, and your handler, meaning the code sitting at that URL to receive the event, must verify that signature before believing a single field [6]. Skipping verification is the webhook equivalent of accepting a check without confirming it's real.
The second non-negotiable is idempotency: your handler must be safe to receive the same event twice. Providers retry delivery when your server is slow or briefly down, so duplicates are normal, not exceptional [6]. A handler that credits an account on every 'payment succeeded' event will double-credit on a retry, a real, money-losing bug. The fix is to record each event's unique ID and ignore one you've already processed, so the second delivery is a harmless no-op.
7.4 Payment integrations
Taking real money is where sloppy integration stops being a bug and starts being a liability, so payment providers like Stripe are deliberately opinionated about how it's done. The standard flow has three moves: collect card details on the client using the provider's hosted checkout or pre-built elements, create the charge or subscription on your server, and confirm the outcome via a webhook. Each move exists for a reason worth understanding rather than copying blindly.
Card details go through the provider's UI specifically so raw card numbers never touch your server. Handling raw card data yourself drags you into serious compliance obligations (PCI) you almost certainly don't want. Letting Stripe's elements capture the card means the sensitive data goes provider-to-provider, and your server only ever sees a safe token.
Confirm success via webhook, not via the user's browser redirect, because the browser is unreliable at the worst moment. A customer can pay and then close the tab, lose signal, or have their phone die before the redirect fires. If that redirect was your only signal, your server never learns the payment happened and never ships the goods. The webhook (with the verification and idempotency from the last section) is the trustworthy channel, leaving the redirect as a nicety for the user and nothing more.
Stripe is the default this book uses because its documentation and developer tooling set the industry standard. It is one player among many payment providers, and knowing the cast helps you pick deliberately. PayPal remains the most widely recognized checkout button and still lifts conversion with buyers who don't want to type a card (Braintree is its developer-oriented arm). Square dominates when a business also sells in person, since one system handles the card reader and the website, while Adyen processes at massive scale for global enterprises, though a newer category matters more if you're selling software. Merchant-of-record providers like Paddle and Lemon Squeezy legally act as the seller, which means they handle sales tax and VAT across every country for you. For a solo developer selling a $10 tool worldwide, the merchant-of-record premium is real, since these providers take a few percentage points more per sale than a plain processor does. The tax handling is usually still the better deal, because registering for and remitting sales tax in every country you sell into costs far more time and money than the fee gap.
Two practical notes on choosing. First, fees across the plain processors cluster tightly enough that for a course project the difference is noise. Pick on developer experience, on what your platform integrates natively (many app builders and site platforms have one-click Stripe integration), or on merchant-of-record tax handling if you're selling internationally. Second, everything this chapter taught about the integration is processor-agnostic: hosted checkout keeps card data off your server, the webhook is the source of truth, handlers verify signatures and stay idempotent, and prices live server-side. Swap Stripe for any provider in the table, and the architecture doesn't change.
And the rule that ties the whole chapter's security thread together: never trust an amount sent from the client. A price submitted in a form or request body can be altered by anyone (change $49.99 to $0.01 and resubmit), so the source of truth for what something costs must live on your server or with the provider, looked up by product ID, never read from the request. Treating client input as untrusted is a theme you'll see again in Chapter 14. Payments are where ignoring it costs actual money.
Provider
Best known for
Merchant of record?
Pick it when
Stripe
Developer experience, docs, and pre-built checkout/elements
Not by default, though Stripe Managed Payments adds merchant-of-record selling where you qualify
Default choice for custom apps; the one this book uses
PayPal / Braintree
The most recognized checkout button
No
Your buyers trust the PayPal button more than a card form
Square
Unified in-person + online payments
No
The business also sells face-to-face with a card reader
Paddle
Merchant of record for SaaS and software
Yes, it handles global sales tax and VAT as the legal seller
Selling software internationally without a tax department
Lemon Squeezy
Merchant of record for indie digital products, owned by Stripe since 2024
Yes
A solo dev selling digital products worldwide
Adyen
Enterprise-scale global processing
No
Large-scale, multi-country operations
Summary
Real apps talk to other services. We covered REST APIs, calling AI APIs, webhooks, and payments, including a snapshot of processors in use (Stripe, PayPal, Square, Adyen) and the merchant-of-record category (Paddle, Lemon Squeezy) that handles global sales tax by legally acting as the seller. The recurring theme: AI makes the call easy but doesn't make an API's quirks go away. Payment architecture stays processor-agnostic: hosted checkout, webhook as source of truth, server-side prices. We also called out a vibecoding-specific risk. AI tools default to whatever gets code running, which often means hardcoding a key, so you insist on the safer route yourself and rotate immediately if one ever leaks. The durable habits are keeping keys server-side, verifying webhook signatures, and making handlers idempotent. Then we handle rate limits and errors instead of crashing on them.
Key terms
REST API. A way of exposing operations over HTTP organized around resources (GET /users/42 reads a user, and POST /users creates one).
Endpoint. A specific URL and HTTP method an API exposes for one operation.
API key. A secret credential that authenticates your app to a service. It must stay server-side, never shipped in client code.
Rate limit. A cap on how many requests you can make in a time window. Exceeding it returns an error your code must handle.
Streaming. Receiving a model's response token-by-token as it's generated, enabling a typing-style UI instead of waiting for the full reply.
Webhook. An event another service pushes to a URL you provide when something happens, the reverse of you requesting data.
Idempotency. A property where performing the same operation twice has the same effect as once, essential for safely handling retried webhooks.
Polling. Repeatedly asking a service 'has anything changed yet?' An alternative to webhooks that is simpler but less efficient.
Environment variable. A configuration value (like an API key) supplied to your app at runtime rather than hardcoded in source.
Secret leak. A credential (API key, password, token) exposed somewhere it can be read by someone who should not have it: a git commit, a public repo, a pasted chat log, or a screenshot. Rotate the credential immediately once one happens; don't wait to confirm it was actually misused.
Signature verification. Checking a cryptographic signature on a webhook payload to confirm it really came from the expected sender.
Stripe. The payment processor this book uses by default: card details go through its hosted checkout or elements, your server creates the charge, and a webhook confirms the outcome. Its documentation and tooling set the industry standard.
Payment processor. A service (Stripe, PayPal, Square, Adyen) that handles card collection, charging, and payout so your app never touches raw card data.
Merchant of record. A provider that legally acts as the seller of your product, taking on global sales tax and VAT obligations in exchange for a higher fee. Paddle and Lemon Squeezy are built around this model, and Stripe offers it as an add-on called Managed Payments.
PCI compliance. The security obligations that come with handling raw card data, the burden that hosted checkouts exist to keep off your server.
Check your understanding
Q1. Where should an AI provider's API key live in a web app?
On the server only, never sent to the browser
In client-side JavaScript so the browser can call the API directly
It doesn't matter where it lives
Hardcoded into the frontend build for convenience
Answer: A. A key visible in browser code is visible to anyone who opens dev tools, and AI API usage is billed, so an exposed key is a financial risk.
Q2. You just discovered an API key was accidentally committed to a public GitHub repo three weeks ago, and you've since removed it from the latest commit. What should you do?
Nothing further, since it's gone from the current version of the code
Rotate the key now, since it's still readable in the commit history
Make the repository private and leave the existing key in place
Wait and watch for unusual usage before deciding whether to act
Answer: B. Deleting a key from a later commit doesn't remove it from git history, and a public repo is actively scanned by bots within minutes. Once a key is exposed, treat it as compromised and rotate it right away.
Q3. Why should webhook handlers be idempotent?
Idempotency is a database concern and doesn't apply to webhooks
Webhook delivery is guaranteed once and is never retried
Providers retry delivery, so the same event can arrive more than once
It measurably improves the throughput of the handler
Answer: C. If your handler isn't safe to run twice on the same event, a routine retry can cause duplicate side effects like double-charging a credit.
Q4. What does a 'merchant of record' provider like Paddle or Lemon Squeezy do that a plain processor like Stripe doesn't?
It offers lower per-transaction fees than every other processor
It lets you store raw card numbers on your own server safely
It removes the need to handle webhooks in your application
It acts as the legal seller and handles global sales tax for you
Answer: D. A merchant of record is the legal seller, so tax obligations across countries land on it instead of you, often the deciding factor for a small team selling software internationally.
Q5. Why should you never trust a payment amount sent from the client?
Client code can be edited to send any value, so the server must decide
Browsers are technically incapable of sending numeric values
Trusting the client is fine as long as the user is signed in
Payment amounts are always validated automatically by the browser
Answer: A. A user (or a script) can alter client-side requests, so pricing must always be looked up or verified server-side.
Practice
Exercise 1. Pick a public REST API with a free tier. Make an authenticated request, then deliberately exceed its rate limit and observe what error it returns. (Hint: Read the documented error shape before you trigger it. Then confirm your handling actually matches what came back.)
Exercise 2. Ask an AI tool to 'add a call to [any AI API] from my app' with no other instruction, and see where it puts the key. Then ask it to fix it so the key is server-side only, via an environment variable, and never appears in any file you'd commit. (Hint: Check the diff carefully. It's common for a key to still leak into a comment, a console.log, or a config file that isn't in .gitignore even after the 'fix.')
Exercise 3. Design (don't necessarily build) a webhook handler for a payment-success event. Write down what you'd check before trusting the payload and how you'd make the handler safe to receive twice. (Hint: Signature verification answers 'is this really from the provider?' Idempotency answers 'what if I get this exact event again?')
Exercise 4. You're a solo developer about to sell a $12 desktop app to customers in 20 countries. Using the processor table, pick a provider and defend the choice in a paragraph: what did merchant-of-record status, fees, and developer experience each contribute to the decision? (Hint: Ask who is legally responsible for collecting VAT on a sale to Germany under each option. That question usually decides it.)
Sources
Orca Security, 2023 Honeypotting in the Cloud Report: exposed GitHub keys found and used within 2 minutes (June 2023) (https://orca.security/resources/blog/2023-honeypotting-in-the-cloud-report/)
Unit 42 (Palo Alto Networks), EleKtra-Leak: exposed IAM credentials on GitHub used within about 5 minutes (October 2023) (https://unit42.paloaltonetworks.com/malicious-operations-of-exposed-iam-keys-cryptojacking/)
GitGuardian, The State of Secrets Sprawl 2026: 28.6M new secrets leaked on public GitHub in 2025, with AI-service credentials up 81 percent (https://www.gitguardian.com/state-of-secrets-sprawl-report-2026)
GitHub docs: About secret scanning (https://docs.github.com/en/code-security/secret-scanning/introduction/about-secret-scanning)
GitHub docs: Removing sensitive data from a repository: history persists, rotate the credential first (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository)
Stripe docs: webhook signature verification and duplicate-event handling (https://docs.stripe.com/webhooks)
8. Agents and Automation
Multi-step reasoning, orchestration, research assistants, and always-on personal bots.
Agents turn a single prompt into a sequence of actions toward a goal. This chapter covers how agents plan and use tools, how they chain steps into reliable automation, and what always-on personal agents (Clawdbot / OpenClaw and friends) change when the loop never really stops.
By the end of this chapter, you will be able to
Explain the plan, act, observe loop that distinguishes an agent from a single model call
Apply guardrails (step limits, validation, retries) to prevent a runaway agent loop
Decide where a human checkpoint belongs based on stakes and reversibility
Build a simple research-assistant agent that cites its sources
Describe what an always-on personal agent (Clawdbot / OpenClaw) is: gateway, channels, skills, and where checkpoints still belong
Explain how computer-use agents work (screenshot in, action out), when pixels are the right interface, and why the screen itself is an attack surface
8.1 What makes something an agent
Everything so far has been mostly single-turn: you ask, the model answers, and you decide what to do next, whereas an agent moves that decision-making inside the loop, pursuing a goal across many steps instead of answering one question. It decides on an action (usually a tool call), takes it, observes the result, and uses what it learned to decide the next action, repeating until the goal is met or it gives up [1][5].
The crucial thing to understand is that the model itself isn't new or special. It's the same language model from earlier chapters, with scaffolding added around it: a loop, access to tools, and somewhere to keep track of progress, so 'agent' names an architecture rather than a different brain. This demystifies the term: when you hear a product is 'agentic,' it usually means a familiar model wrapped in a plan, act, observe loop with tools attached. An agent, in other words, is a model in a loop. An unsupervised agent is a loop with your credit card.
That loop is what lets agents tackle problems too big for one shot. 'Find the three cheapest flights and book the best one' isn't a single answer. It's search, compare, decide, act, each step depending on the last. The agent decomposes the goal as it goes, which is powerful precisely because it can adapt: if a search returns nothing, it can try a different query rather than failing the whole task.
It's also the source of agents' characteristic risk. A single wrong answer is contained. A wrong decision early in a loop compounds, because every later step builds on it. Much of this chapter is about the structures (reasoning steps, guardrails, checkpoints) that keep that compounding in check, which is why agents demand more design discipline than a one-off prompt does.
What turns a chatbot into an agent: a loop that can use tools, observe the result, and decide what to do next.
Plan โ Act โ Observe โ (repeat)
The plan, act, observe loop that turns a model into an agent. It decides an action, takes it, observes the result, and repeats until the goal is met or it gives up.
8.2 Multi-step reasoning
An agent makes better decisions when it reasons before it acts, instead of lunging at the first plausible action. The pattern most frameworks use is to interleave an explicit 'thought' step with each action. The model writes out what it currently knows, what it still needs, and which action would close that gap, and only then calls a tool. This is the same chain-of-thought idea from Chapter 2, now driving behavior instead of just producing an answer.
The popular name for this is ReAct (reason + act) [2], and the reason it helps is mechanical: forcing the model to articulate its plan gives it a chance to notice a contradiction or a missing piece before committing to an action it can't take back. A person muttering 'wait, I already checked that' under their breath is doing the same error-catching, and writing the thought out gives the model that same beat.
These thought traces are also a gift to you as the supervisor. Because the agent narrates its reasoning, you can watch where its logic went wrong, instead of only seeing a bad final result. The moment it misread a tool's output, or assumed something false, is right there in the trace. When an agent run fails, reading the thoughts is usually the fastest way to find the step that derailed.
The caveat from Chapter 9 still applies: a stated reason isn't a guarantee of a correct one. A model can produce confident, well-formed reasoning that's wrong, and then act on it just as confidently. Research on this finds that a model's stated reasoning isn't always a faithful account of what actually drove its answer [3], so the thought step improves the odds and aids debugging. It doesn't make the agent trustworthy on its own, which is exactly why the next sections add hard constraints around the reasoning rather than relying on it.
8.3 Workflow orchestration
Orchestration is how you compose agents and tools into a reliable pipeline instead of hoping a single agent figures everything out. A common shape is a sequence of stages (one gathers data, another summarizes it, another formats the result), each with a narrow, well-defined job. Sometimes this is several specialized agents (a multi-agent system, e.g. a planner that delegates to workers) [4]. Often it's simpler to hard-wire the steps you already know need to happen and only use an agent where genuine open-ended decision-making is required. Orchestration here means composing agents into a system you build. Deciding whether to point several agents at your own codebase at the same time is a different question, covered in 3.7.
That last point is a real design principle: not everything that could be an agent should be. If you know the steps in advance, a plain scripted pipeline is more predictable, cheaper, and easier to debug than handing the whole thing to an autonomous agent [1]. Reserve agentic autonomy for the parts that are actually open-ended, and keep firm rails around the parts that aren't.
Reliable orchestration leans on guardrails. Validate a stage's output before passing it downstream (so one bad result doesn't silently poison everything after it). Retry steps that fail transiently (a flaky network call) instead of aborting the whole run. And cap how many steps or how much money an agent may spend. These turn a fragile demo into something you'd actually trust unattended.
The problem guardrails prevent has a name: the runaway loop. Without a step limit, an agent that isn't converging will keep calling tools and keep billing you, sometimes for a long time on a problem it was never going to solve. A maximum step count and an explicit 'give up and report' path aren't nice-to-haves. They're the difference between an agent that's safe to leave running and one that quietly empties your API budget overnight.
8.4 Human checkpoints
Autonomy is a dial, not a switch, and human checkpoints are how you set it. A checkpoint is a deliberate pause where the agent stops and waits for a person to review or approve before continuing, such as before sending an email, charging a card, deleting a record, or merging code, with the agent doing the work up to that line and a human deciding whether to cross it.
Where to place checkpoints follows from two questions: how high are the stakes, and how reversible is the action? Drafting a summary for you to read needs no checkpoint, because if the summary is wrong, you simply don't use it. Sending that summary to 500 customers, or deleting production data, needs a checkpoint every time, because the cost of being wrong is high and you can't take it back. Reversibility is the key axis: automate freely where mistakes are cheap to undo, and gate hard where they aren't.
This connects straight to Chapter 10's point about human oversight and Chapter 14's about responsible deployment: the goal is appropriate automation, not maximum automation. An agent that asks before every trivial action is annoying and trains people to rubber-stamp. One that asks before nothing is dangerous. Designing the right checkpoints, frequent where it counts and absent where it doesn't, is a genuine product decision.
A practical pattern is graduated trust: start a new agent with many checkpoints while you learn the ways it goes wrong, then remove the ones that consistently prove unnecessary as you build confidence. You earn autonomy for the agent the same way you'd earn it for a new employee, by watching it handle the low-stakes cases well before handing it the high-stakes ones.
8.5 Building a research assistant
A research assistant is the canonical first agent because it exercises every idea in this chapter on a task you can actually judge. The basic loop: take a question, search for sources (a tool call), read and extract the relevant content from each result, decide whether that's enough, and synthesize a cited summary, where each step is simple on its own and the intelligence is in the sequencing and the stopping.
Counterintuitively, the search step is rarely the hard part. Calling a search API is easy. The hard parts are judgment calls: knowing when enough sources have been gathered (stop too early and you miss the key one, too late and you waste steps and tokens), and noticing when two sources disagree instead of blending a contradiction into a confident, wrong sentence. These are exactly the decisions a naive agent gets wrong, and where your design (what to check, when to stop) earns its keep.
Citation is the non-negotiable feature that makes the whole thing trustworthy. A summary that asserts facts without sources is just the hallucination problem from Chapter 9 wearing a tie, authoritative and unverifiable. An assistant that attaches the specific source behind each claim turns its output from 'trust me' into 'check for yourself,' which is the only honest way to ship something built on retrieved information.
Build one, and you'll feel why everything earlier in the chapter exists. You'll want a step limit the first time it loops on a bad query, a guardrail the first time a tool returns garbage, and a checkpoint the first time you imagine it emailing its summary somewhere automatically. The research assistant is small enough to finish in a lab and rich enough to teach the discipline that larger agents demand.
8.6 Clawdbot, OpenClaw, and always-on personal agents
Most of this chapter treats an agent as something you start for a task and stop when the task is done: a research run, a coding ticket, a pipeline stage. Always-on personal agents invert that arrangement entirely. The agent lives as a long-running gateway on your machine (or a server you control), and you talk to it in the apps you already use: WhatsApp, Telegram, Discord, Slack, Signal, iMessage, and a long list of others. The product people still casually call 'Clawd bot' is that idea made concrete.
OpenClaw (formerly Clawdbot, and for three days in January 2026 Moltbot) is an open-source personal AI assistant created by Peter Steinberger [6][8][11]. The OpenClaw Foundation, announced in February 2026 and formally launched on July 8, 2026, now maintains it as a nonprofit [22]. The rename followed an Anthropic trademark issue, alongside reporting on hundreds of exposed instances left open on the public internet [9][12]. You run a local-first Gateway as the control plane for sessions, channels, tools, and events. The model is still the familiar brain from earlier chapters. What changed is the scaffolding: persistent memory, multi-channel routing, companion apps, and a skills system so the assistant can browse, draft email, touch GitHub, control notes, or grow new capabilities without you rewriting the whole agent.
Skills are the 'and stuff' that make the lobster useful day to day. A skill is typically a folder with a SKILL.md that teaches the agent how to use a tool or workflow [7]. Bundled skills ship with the install; workspace and registry skills (ClawHub / community packs) extend it. The agent can even help author a skill when you ask for a capability it doesn't have yet. That's vibecoding applied to the assistant itself: you grow the bot by talking to it, then pin the result as reusable scaffolding.
OpenClaw belongs in the core agents chapter because it is the plan, act, observe loop living in your pocket, and the same risks scale with the surface area. A wrong early tool call can still compound, and a runaway loop can still empty an API budget overnight. A message that looks like a normal chat can still carry a prompt injection that hijacks the agent (Chapter 14). The difference is that the damage now reaches whatever channels and credentials you connected.
Design the autonomy dial deliberately. Drafting a reply in Telegram might need no checkpoint. Sending mail to your whole class, spending money, deleting files, or installing a fresh skill from the internet needs a pause you actually review. Start narrow: one channel, one model provider, a short allowlist of skills, step limits on tool use. Earn wider access the same way Section 8.4 earns autonomy for any new agent: watch the low-stakes cases, then remove the checkpoints that consistently prove unnecessary.
There's also a human lesson wrapped around the technical one. Steinberger has talked publicly about agentic coding becoming an addiction. He opened one talk with 'Hi, my name is Peter and I'm a Claudoholic' and compared coding agents to slot machines [10]. The assistant is so available that you keep pulling the lever instead of living. None of that argues against personal agents, only for treating them like power tools with a schedule, using them to shrink boring loops without outsourcing every idle minute to another session.
Same product, several namesPeople still say 'Clawdbot' or 'Clawd bot' in chat. It's the same project as OpenClaw, renamed twice on the way.
Always-on means always attackableA gateway that can read your inbox, browse the web, and run tools from a DM is powerful because it's connected. This isn't hypothetical: in early 2026 a security researcher found hundreds of Clawdbot-era gateways exposed to the open internet, some with no authentication at all, and malicious skills on the registry have been caught exfiltrating credentials [23]. Current versions bind to localhost and require token auth by default, but the lesson stands. Treat channel access, API keys, and skill installs like production credentials, and pair high-stakes actions with human approvals the same way you would for any other agent.
Try the onboard path before you invent oneOpenClaw's recommended start is the CLI onboard flow (install, then openclaw onboard --install-daemon). It walks gateway, workspace, channels, and skills. You learn more from one working DM loop than from reading about twenty unconfigured ones.
8.7 Computer-use agents with your mouse and keyboard
Every agent so far has acted through structured interfaces: an API to call, a shell to run, a file to edit. Computer use removes that requirement. The model takes a screenshot, reasons about what it sees (the same vision capability Section 11.6 uses to read the screenshots you hand it), and issues clicks, keystrokes, and scrolls, the same loop from 8.1 with the world's most general and least reliable tool attached. All three major labs now ship a version. Anthropic's computer-use tool (in beta since October 2024) [13] powers a research preview in Claude Code and the desktop app [14]; OpenAI folded its Operator experiment into ChatGPT's agent mode, then retired agent mode in July 2026 and moved that work into ChatGPT Work, which breaks long jobs into steps and runs them on OpenAI-hosted infrastructure, while the ChatGPT desktop app on macOS and Windows carries screen control [15][16]; Google's Gemini computer-use models started browser-first and now cover browser, desktop, and mobile environments [17][24]. The differences matter less than the shared shape: screenshot in, action out, repeat.
Why reach for pixels when Chapter 4 gave you tools and Chapter 7 gave you APIs? Because some software has no other interface. Legacy desktop applications, GUI-only admin panels, a phone simulator, the settings dialog of a native app: for these, the screen is the only door. That makes computer use the interface of last resort, and the vendors say so themselves. Anthropic's own documentation ranks the options [14]: use an MCP server or API when one exists, the shell when the task is shell-shaped, a browser extension for browser work, and screen control only for what nothing else can reach. Every layer you can replace with a structured interface buys back speed, cost, and reliability, because each screen action is a full model round-trip over a screenshot. The same task that takes an API call milliseconds can take a computer-use agent minutes, at vision-token prices.
The honest capability picture has two numbers in it. On OSWorld, the standard benchmark of real desktop tasks, agents went from about 15 percent success in late 2024 to passing the human baseline of roughly 72 percent during 2026, with the best published runs now above 85 percent [18], which is why the benchmark had to be replaced. Its successor, OSWorld 2.0, measures hour-plus professional workflows instead of minutes-long tasks, where the best model completes about 21 percent [19]. Both numbers are true at once, and together they're the lesson: computer use is genuinely reliable for short, well-scoped GUI work like clicking through an interface to reproduce a bug, filling a form, or driving a simulator, while it still fails most tasks that would take a person an hour. Demos live in the first regime, so plan as if your task might live in the second.
Security deserves its own paragraph, because the attack surface here is the screen itself. A computer-use agent reads everything it sees, which means a webpage, an email, or an image can contain instructions planted for the agent, Chapter 14's prompt injection with a monitor for a delivery vehicle. Researchers have already demonstrated a browser agent being steered by a hidden instruction in a Reddit comment into extracting its user's email address and one-time code [20]. The vendors' mitigations are worth knowing because they're also a checklist for your own agent designs: classifiers that scan screenshots for planted instructions, per-app permission tiers (Claude's desktop preview caps browsers and trading platforms at view-only, terminals and IDEs at click-only), confirmation prompts before consequential actions, takeover modes where you type credentials the model never sees, and a denied-apps list you fill in yourself. Anthropic's published red-teaming is candid that none of this is complete: mitigations cut attack success rates sharply, but not to zero on the open web [21].
So the practical stance for a vibecoder is the same autonomy dial from 8.4, set one notch more conservative. Run computer-use agents in an environment you can afford to lose, a virtual machine or a spare user account, not your logged-in daily machine. Give them tasks with checkable endpoints, keep credentials out of reach, and leave confirmations on. And before reaching for the mouse at all, ask the Chapter 4 question: is there a tool for this? The best computer-use session is often the one you replaced with an API call.
Offering
Scope
Status (mid-2026)
Anthropic computer-use tool (API)
Full desktop, client-side: your code runs the actions
Beta
Claude Code / Claude Desktop computer use
macOS (CLI), macOS and Windows (desktop app), per-app permission tiers
Research preview, Pro/Max plans
ChatGPT Work (successor to agent mode)
Browser and terminal on OpenAI-hosted infrastructure, plus desktop screen control in the ChatGPT app
Shipped July 2026; agent mode absorbed Operator and was then retired
ChatGPT/Codex desktop computer use
macOS and Windows native apps, summoned by @-mentioning an app
Shipped
OpenAI computer tool (API)
Browser or full OS, your harness executes actions
GA on GPT-5.x models
Gemini computer-use models (API)
Browser-first, now with desktop and mobile environments
Preview
The screen is an input channelAnything a computer-use agent can see can instruct it: a webpage, an email, an image with text in it. Treat every screenshot as untrusted input, run the agent somewhere disposable, and keep confirmation prompts on for anything that moves money, sends messages, or deletes data. Vendor classifiers help and aren't sufficient.
Pixels lastBefore giving an agent the mouse, walk the hierarchy: is there an API? A CLI? An MCP server? A browser extension with DOM access? Screen control is the slowest, priciest, and most attackable option, so it should win only when nothing structured exists. This ordering comes straight from the vendors' own docs.
Summary
An agent is a familiar model wrapped in a plan, act, observe loop with tools, not a different kind of brain. That loop lets it tackle multi-step goals. But a wrong early decision compounds, so the design work is in reasoning steps, guardrails, step limits, and human checkpoints sized to the stakes. Always-on personal agents like OpenClaw (the project formerly called Clawdbot) show what happens when that loop lives in your messaging apps with skills and a local gateway. Reach for an agent when a task is genuinely multi-step. Script it when it isn't. Computer use extends the same loop to the screen itself: reliable for short, well-scoped GUI tasks, still failing most hour-long workflows, and best treated as the interface of last resort.
Key terms
Agent. An AI system that pursues a goal across multiple steps, choosing actions, observing results, and continuing until done.
Plan-act-observe loop. The core agent cycle: decide on an action, take it, observe the outcome, and repeat.
ReAct. A common agent pattern interleaving reasoning ('thought') steps with actions, so the model can sanity-check before acting.
Tool use. An agent's ability to call functions, APIs, or other systems to gather information or take action beyond generating text.
Orchestration. Composing multiple agents and tools into a reliable pipeline with validation, retries, and ordering.
Multi-agent system. An arrangement where several specialized agents collaborate, e.g. a planner that delegates to worker agents.
Guardrails. Constraints on an agent (step limits, output validation, allowed actions) that keep it safe to run with less supervision.
Human checkpoint. A deliberate pause where a person reviews progress before the agent continues, used before high-stakes or irreversible actions.
Autonomy. The degree to which an agent is allowed to act without a human checkpoint, a dial, not a switch, earned gradually by watching it handle low-stakes cases well before removing checkpoints on higher-stakes ones.
Runaway loop. An agent that keeps acting without converging on its goal, burning time and API cost, which step limits prevent.
Scaffolding. The surrounding code and structure (memory, tools, control flow) wrapped around a model to turn it into an agent.
Clawdbot. The name the open-source personal AI assistant carried through January 2026, after two earlier names and before a brief stretch as Moltbot and the rename to OpenClaw. People still say 'Clawd bot' in conversation.
OpenClaw. A self-hosted personal AI assistant: a local-first gateway that connects messaging channels to an always-available agent with tools, memory, and skills.
Gateway. The long-running control plane process that owns channel connections, sessions, tools, and events for a personal assistant like OpenClaw.
Computer use. An agent capability where the model takes screenshots and issues clicks, keystrokes, and scrolls to operate real software, the plan, act, observe loop with the screen as its tool.
Interface of last resort. The vendors' own ranking for computer use: prefer an API, CLI, MCP server, or browser extension when one exists, and control the screen only when nothing structured can reach the task.
Takeover mode. A safety pattern where the human takes over the session to type credentials or pass checkpoints, so the model never sees the sensitive input.
Skill. A packaged capability (often a SKILL.md plus supporting files) that teaches an agent how to use a tool or workflow. Skills can be bundled, workspace-local, or installed from a registry.
Always-on agent. An agent that stays running and reachable (usually via chat apps) instead of starting only when you open an IDE session for a single task.
Check your understanding
Q1. What distinguishes an agent from a single model call?
Agents use a fundamentally different, more powerful model
It pursues a goal over many steps in a plan, act, observe loop
Agents operate without access to any external tools
There's no meaningful difference beyond the marketing term
Answer: B. The model is the same. The loop wrapped around it (deciding, acting, observing, repeating) is what makes it an agent.
Q2. Why are guardrails like a maximum step count important for agents?
They meaningfully reduce the latency of every single agent step
Guardrails only matter for multi-agent systems, not single ones
Without them, an agent can run on, burning money on a problem it can't solve
They're cosmetic and make no real difference to reliability at all
Answer: C. An unbounded loop on a problem the agent isn't solving is the classic 'runaway loop' failure. Guardrails cap the damage.
Q3. When should you insert a human checkpoint into an agent's workflow?
Never, because full automation is always the goal to aim for
Only once, at the very start of the workflow before it runs
After every single tool call, regardless of what's at stake
Before high-stakes or hard-to-reverse actions, scaled to the stakes
Answer: D. Checkpoint placement should track risk and reversibility. A draft summary needs none, while deleting production data needs one every time.
Q4. What's OpenClaw (often still called Clawdbot) primarily?
A self-hosted gateway connecting chat channels to an always-on agent
A new foundation model built to replace Claude and GPT entirely
A hosted chatbot that runs only inside a code editor window
A continuous integration service that only runs unit tests
Answer: A. OpenClaw is scaffolding around models you already use: a local-first gateway, messaging channels, memory, tools, and installable skills, not a brand-new brain.
Q5. You want an agent to update a value in a legacy desktop app that has no API, no CLI, and no browser version. A friend suggests a computer-use agent. What's the right way to think about that choice?
Never reach for computer use, since it's always the wrong tool
Right call, since nothing structured exists, but run it in a virtual machine or a spare account, not your daily machine
Prefer computer use over APIs generally, since it works on anything
Run it on your daily machine so the agent has your credentials
Answer: B. Pixels are the interface of last resort: slow, expensive, and attackable, but the only door into GUI-only software. When nothing structured exists, computer use earns its place, and the safeguards (disposable environment, checkable endpoint, confirmations, no credentials in reach) are what make it survivable.
Q6. OSWorld shows agents past the ~72% human baseline on real desktop tasks, with top runs above 85%, while OSWorld 2.0 shows the best model completing about 21% of hour-plus workflows. What should you conclude?
The two benchmarks contradict each other, so one of them is wrong
Agents have now surpassed humans across all computer-based work
They're reliable on short, scoped GUI tasks and fail most long ones
Benchmarks are meaningless for judging real agent capability
Answer: C. Both numbers are true at once: the original benchmark's minutes-long tasks are effectively solved and its human baseline passed, which is why it was replaced, and its successor's hour-plus workflows mostly aren't. Demos live in the first regime; plan as if your task might live in the second.
Practice
Exercise 1. Design (on paper) a research-assistant agent for answering 'what are the top 3 trade-offs of approach X'. List its steps, where it calls tools, and where you'd insert a human checkpoint. (Hint: Identify the one step most likely to introduce an error (e.g. a low-quality source) and decide whether a checkpoint or a guardrail handles it better.)
Exercise 2. Take an agent framework's default step limit and explain, in your own words, what would go wrong without it on a task the agent can't actually solve. (Hint: Think in terms of cost and time, not just correctness. An agent that never gives up is still failing.)
Exercise 3. Sketch a minimal OpenClaw / Clawdbot setup for yourself: which one messaging channel, which three skills, and which two actions would always require a human checkpoint. Explain why each checkpoint exists in terms of stakes and reversibility. (Hint: Prefer boring, reversible skills first (drafting, summarizing) over anything that sends, spends, deletes, or installs.)
Exercise 4. Pick a small GUI-only task on your own machine (change a setting in a native app, reproduce a visual bug by clicking through an interface). First write down the pixels-last check: is there actually an API, CLI, MCP server, or browser path for it? If there truly isn't, run it with a computer-use agent in a safe environment and record how many actions and how long it took versus doing it by hand. (Hint: The write-up matters more than the run. Most students discover mid-exercise that a structured interface existed after all, which is itself the lesson. If you do run it, watch the screenshot-action loop and note the first place the agent misreads the screen.)
Sources
Anthropic, Building Effective Agents: workflows vs. agents, guardrails, stopping conditions, and checkpoints (December 2024) (https://www.anthropic.com/engineering/building-effective-agents)
Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models (ICLR 2023) (https://doi.org/10.48550/arXiv.2210.03629)
Turpin et al., Language Models Don't Always Say What They Think: plausible but unfaithful reasoning (NeurIPS 2023) (https://doi.org/10.48550/arXiv.2305.04388)
How we built our multi-agent research system (lead agent delegating to subagents) (https://www.anthropic.com/engineering/multi-agent-research-system)
How tool use works: the agentic loop of tool calls until the goal is met (https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works)
OpenClaw, the personal AI assistant (product site) (https://openclaw.ai/)
OpenClaw docs: getting started, gateway, skills, and onboarding (https://docs.openclaw.ai/)
openclaw/openclaw on GitHub (formerly clawdbot/clawdbot) (https://github.com/openclaw/openclaw)
CNET, From Clawdbot to Moltbot to OpenClaw (naming history and product overview) (https://www.cnet.com/tech/services-and-software/from-clawdbot-to-moltbot-to-openclaw/)
Steinberger, Just One More Prompt: agentic coding as addiction (steipete.me, August 2025) (https://steipete.me/posts/just-one-more-prompt)
OpenClaw blog: Introducing OpenClaw, the rename announcement (January 2026) (https://openclaw.ai/blog/introducing-openclaw)
The Register: hundreds of exposed Clawdbot instances, and the Anthropic trademark rename (January 2026) (https://www.theregister.com/2026/01/27/clawdbot_moltbot_security_concerns/)
Anthropic docs: the computer-use tool, capabilities, limitations, and prompt-injection warnings (https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool)
Claude Code docs: computer use research preview, permission tiers, and the tool hierarchy (https://code.claude.com/docs/en/computer-use)
OpenAI docs: the computer tool for browser and full-OS control (https://developers.openai.com/api/docs/guides/tools-computer-use)
Google DeepMind, the Gemini 2.5 Computer Use model (October 2025) (https://blog.google/technology/google-deepmind/gemini-computer-use-model/)
Xie et al., OSWorld: benchmarking real computer tasks (the ~72% human baseline) (https://osworld-v1.xlang.ai/)
OSWorld 2.0: hour-plus professional workflows, where the best model completes about 21 percent (June 2026) (https://osworld-v2.xlang.ai/)
Brave security research: indirect prompt injection in the Comet browser agent (August 2025) (https://brave.com/blog/comet-prompt-injection/)
Anthropic, Claude for Chrome pilot: measured prompt-injection attack rates with and without mitigations (https://claude.com/blog/claude-for-chrome)
OpenClaw blog: Introducing the OpenClaw Foundation, the nonprofit steward launched in July 2026 (https://openclaw.ai/blog/introducing-openclaw-foundation)
Unit 42, OpenClaw's skill marketplace and the emerging AI supply chain threat: malicious skills exfiltrating credentials (June 2026) (https://unit42.paloaltonetworks.com/openclaw-ai-supply-chain-risk/)
Gemini API docs: the computer use tool across browser, desktop, and mobile environments (https://ai.google.dev/gemini-api/docs/computer-use)
9. AI Slop and How AI Goes Wrong
Hallucinations, 'looks right' syndrome, and overreliance.
The course's reality check. AI is confidently wrong in patterned ways. This chapter catalogs the ways AI goes wrong (hallucination, misinformation, SEO and coding slop, 'looks right' syndrome) and builds the instinct to distrust and verify.
By the end of this chapter, you will be able to
Define hallucination and explain why fluency isn't evidence of correctness
Recognize SEO slop and coding slop and explain the incentive structure that produces them
Identify 'looks right' syndrome in AI output before it causes a real bug
Calibrate trust in AI output instead of defaulting to overreliance
9.1 Hallucination
A hallucination is fluent, confident output that's simply wrong [1]: an invented function that doesn't exist in the library, a citation to a paper that was never written, a statistic stated with total certainty that evaporates on a quick search. The unsettling part is the confidence. The model doesn't hedge or signal doubt, because it has no separate sense of 'I'm not sure.' It produces the false claim in exactly the same authoritative tone as a true one. If a friend recommended restaurants this way, you would stop asking.
Understanding why this happens removes the mystery and sharpens your guard. A language model generates the next most plausible piece of text given everything so far. It is, at its core, a very sophisticated predictor of what comes next. Plausible and true are strongly correlated, which is why models are useful at all, but they aren't the same property, and the gap between them is precisely where hallucination lives. The model isn't lying to you, because lying would require knowing better. It has no concept of truth to lie about, only plausibility to optimize. One line of research argues that standard training and evaluation actively reward confident guessing over admitting uncertainty, which is why the behavior persists [2].
Crucially, hallucinations get more dangerous, not less, as a domain gets more technical and specific. An invented API method like `user.getAuthToken()` reads exactly like a real one (same casing, same conventions, same vibe), so nothing about its surface warns you it's fictional. The same goes for whole packages: researchers found code-generating models routinely inventing dependency names that don't exist, which attackers can then register and ship malware under [4]. The more authoritative and specialized the output looks, the more it earns an unearned trust, which is exactly backwards from what's safe.
There's no prompt that fully eliminates this, and that's the chapter's foundational lesson. You reduce hallucination with grounding (Chapter 4's RAG and tool calls give the model real sources instead of memory), and you catch what remains with verification (the rest of this chapter and Chapter 10), but what you never get to do is assume it's gone. Every later problem in this chapter is a variation on 'the plausible thing wasn't the true thing.'
A field guide to AI slop. Each kind fails differently, and the last one, plausible but wrong, is the most dangerous.
Where to stand on trust. The right answer is almost always the right end of this bar.
9.2 From hallucination to misinformation
A hallucination in your private chat is a contained inconvenience. You catch it, or you don't and it costs you an afternoon. The problem changes character entirely when that same false output gets published. At the scale AI makes possible, hallucinations stop being private mistakes and become public misinformation, seeded across thousands of articles, answers, and posts faster than anyone can fact-check them.
What makes this more than ordinary spam is the feedback loop. Search engines index AI-generated falsehoods. Later models train on or retrieve that indexed content. And a fabricated 'fact' can get laundered into apparent credibility simply by being repeated across many AI-written sources that cite each other. A lie with enough citations starts to look like consensus, even when every citation traces back to the same original hallucination.
This is the mechanism behind a worry researchers call model collapse [8]: as more of the web becomes AI-generated, models increasingly learn from other models' output instead of from ground truth, and errors can compound across generations like a photocopy of a photocopy. The open internet that trained today's best models may be harder to learn from tomorrow, precisely because it's filling with synthetic content of uncertain accuracy.
The practical takeaway for you as a builder and a citizen is the same: verify against primary sources, and the more of the web is AI-written, the more that matters. 'I saw it in several places online' is weaker evidence than it used to be, because those several places may share one fabricated origin. Optional Chapter B takes this societal angle further. Here, treat it as the reason your own verification habits matter beyond yourself, a small act of not poisoning the well.
9.3 SEO slop and coding slop
'Slop' is the catch-all term for low-effort, mass-produced AI content that exists to fill space, where usefulness isn't the goal. SEO slop is the web flavor: articles engineered to rank in search results, padded to a great length while saying almost nothing, the recipe site with eight paragraphs of life story before the ingredients, now generated by the thousands. This section finally names something you have been wading through for years.
Coding slop is the software equivalent, and it's sneakier because it can pass a casual check. The code compiles. It looks reasonable. Maybe it even passes a shallow test. But it's needlessly convoluted, copy-pasted into three near-identical variants, or subtly wrong in a way that only surfaces under real load or an unusual input, when a loud failure would have been a gift. Quiet rot is what you get instead, compounding as more of it accumulates.
Both forms exist for one underlying reason worth internalizing, which is that generation is now radically cheaper than verification. It costs almost nothing to produce a plausible article or a plausible function, but real effort to confirm either is actually good. Whenever that asymmetry meets an incentive that rewards volume (ad impressions, a deadline, a velocity metric), slop is the predictable output unless a human actively pushes back on it.
Recognizing slop is a skill you can sharpen, and it's mostly about looking past the surface. Slop tends to lack specific, checkable detail. It restates the obvious. It has the smooth texture of competence without the substance. The defense is to hold generated output to a standard of 'does this actually add correct, specific value' before it goes into your codebase or out to the world.
9.4 'Looks right' syndrome
If this chapter has one named villain, it's this. 'Looks right' syndrome is output that survives a glance (clean syntax, a sensible variable name, a confident grammatical paragraph) but fails the moment you actually run it, test it, or check its claims against reality. The glance says yes while the substance says no, and the gap between them is where most AI-assisted bugs are born.
It's the single most expensive habit to build resistance to, and the reason is structural, not a flaw you can patch. The entire purpose of a fluent model is to produce text that looks right. That's literally the objective it was optimized for. So 'looks right' isn't a signal of correctness at all. It's the default property of all model output, true and false alike. Treating polish as evidence of correctness is reading the one signal the model is guaranteed to fake.
Human psychology makes it worse. We're wired to extend more trust to things that are articulate and confident, so a fluent wrong answer slips past scrutiny that a hesitant, hedged one would trigger. The very smoothness that makes the model pleasant to work with is what disarms the skepticism you most need, a kind of competence theater that your guard has to learn to see through. A controlled study measured this directly: participants with an AI assistant wrote less secure code than those without and were more confident it was secure [3].
The only reliable cure is to replace 'looks right' with 'is proven right' as your bar for trust. Run it (does it behave on real input), test it (does it survive the edge cases of Chapter 10), check it (do its factual claims hold against a source), and, where a mistake would cost money, expose data, or hurt someone, read it (do you actually understand what it does). Each of those is a way of refusing to let appearance stand in for verification, and together they're the working definition of staying in the loop.
Polish isn't correctnessTreat 'looks right' as the starting point for verification, never as a substitute for it.
9.5 Overreliance
Overreliance is the meta-failure that the others feed into: trusting AI output in proportion to how confident it sounds instead of how well it has been verified, which is less a single mistake than a drift. The more the model is right, the more you stop checking, until the one time it's confidently wrong sails straight through because checking is no longer a habit you have. The gap can be invisible from the inside: in one controlled study, experienced open-source developers took longer to finish tasks with AI assistance while believing they had been faster [7]. Models also tend to tell you what you want to hear, agreeing with a confident user even when the user is wrong [6].
Psychologists call the underlying pull automation bias: the documented human tendency to over-trust automated suggestions, sometimes even over our own correct judgment [5], because a confident screen overrides a quiet doubt. With a fluent, almost-always-helpful model, that bias is constantly being trained into you, which is why overreliance creeps up instead of announcing itself, feeling like efficiency right up until it costs you.
Part of what makes the pull so strong is that the model talks like a person. It says 'I think,' it apologizes, and it sounds sure of itself, and it's natural to read those as signs of a mind that knows what it's saying. It helps to remember what is actually there. A language model is a system trained to produce the most plausible continuation of the text it's given, and the confidence, the apology, and the 'I' are all part of that continuation rather than reports from something that checked [9]. Shanahan's advice for talking about these systems is to keep the plain description in mind even while using the convenient shorthand. The model doesn't 'know' the answer. It produces the text that tends to follow questions like yours. The framing doesn't make the tools less useful. It explains why verification has to come from outside the model, since nothing inside it is keeping score.
The fix is calibration, not blanket distrust, and this distinction matters, because cynically second-guessing everything throws away the speed that makes these tools worth using. The goal is to match your trust to the evidence: treat unverified output as a draft no matter how polished it reads, and reserve real trust for output you've actually checked against a test, a source, or a runtime result.
This is ultimately what separates a vibecoder who ships reliable things from one who ships impressive demos that fall apart. The skill the whole book is building toward is calibrated trust: moving fast with AI while keeping a clear, honest sense of what you've confirmed and what you've merely assumed. Chapter 10 turns that mindset into concrete practices (reading, testing, benchmarking) for actually doing the checking.
Summary
This is the course's reality check: AI is confidently wrong in patterned ways. We cataloged hallucination and the slide from hallucination to misinformation. We covered SEO and coding slop, then the central villain: 'looks right' syndrome, where polished output fails the moment you run it. A fluent model is optimized to look right. Polish is the one signal it's guaranteed to fake. The cure: replace 'looks right' with 'is proven right.'
Key terms
Hallucination. Confident, fluent output that is factually wrong, such as an invented API, a fake citation, or a made-up statistic.
Confabulation. A more precise term for hallucination: the model fills a gap with a plausible-sounding fabrication rather than signaling uncertainty.
Misinformation. A hallucination that escapes a private chat and gets published, seeded across articles, answers, and posts faster than anyone can fact-check it, and compounded when other AI-written sources cite it as if it were confirmed.
Sycophancy. A model's tendency to agree with or flatter the user, telling you what you seem to want rather than what is correct.
Looks-right syndrome. Output that survives a glance (correct-looking syntax, confident tone) but fails when actually run, tested, or fact-checked.
SEO slop. Mass-produced, low-quality content generated to rank in search results rather than to inform anyone.
Coding slop. Code that compiles and looks reasonable but is needlessly convoluted, duplicated, or subtly wrong under real use.
Overreliance. Trusting AI output in proportion to how confident it sounds instead of how well it has been verified.
Automation bias. The human tendency to over-trust automated or AI suggestions, even over your own correct judgment.
Model collapse. Degradation that can occur when models are trained on the output of other models, compounding errors over generations.
Anchoring. Being unduly influenced by the model's first answer, making it harder to spot that a different approach was better.
Check your understanding
Q1. Why are hallucinations especially dangerous in technical domains?
Technical domains are immune to hallucination
Models never hallucinate about code, only about facts
Technical hallucinations are always caught by compilers
An invented API reads exactly like a real one, so you skim past it
Answer: D. A plausible-but-fake function name has the same surface texture as a real one, which is exactly what makes it easy to miss without verification.
Q2. What's 'coding slop'?
Code that compiles and looks fine but is convoluted or subtly wrong
Code that fails to compile or run at all
Any code that was written quickly under deadline
Code carrying unusually verbose inline comments throughout
Answer: A. Slop passes a shallow check while hiding problems that only surface under real usage or careful review.
Q3. What's the correct fix for overreliance, according to this chapter?
Distrust all AI output equally, whatever the context happens to be
Match trust to how verified the output is, not how confident it reads
Trust output more whenever it includes citations, whatever they say
Overreliance has no practical fix worth applying
Answer: B. The goal is calibration: treat unverified output as a draft and reserve real trust for output you've actually checked.
Q4. How does a private hallucination turn into public misinformation, according to this chapter?
It doesn't, since hallucinations stay inside the chat that produced them
Search engines automatically fact-check and remove hallucinated content
It gets indexed, then cited across AI-written sources that echo each other
Misinformation only spreads when a human deliberately shares a false screenshot
Answer: C. A false claim published at scale gets indexed and repeated across many AI-generated sources citing each other, laundering a single fabrication into apparent credibility.
Q5. Why doesn't 'I saw it in several places online' count as strong evidence anymore, per this chapter?
It never counted as real evidence, even long before AI existed
Search engines have stopped indexing more than one source per claim
This is a concern only for generated images and never for text
Several sources may all trace back to one original AI hallucination
Answer: D. AI-generated falsehoods can be seeded across many sources that cite each other, so seeing something repeated doesn't mean it was independently verified.
Practice
Exercise 1. Ask a model a moderately obscure factual question and a question about a library function you know well. Try to deliberately induce one hallucination in each. Document what made the false answer convincing. (Hint: Look specifically at phrasing. Hallucinations often use the exact same confident tone as correct answers, which is the point.)
Exercise 2. Find a piece of AI-generated code (yours or someone's) that passed a quick glance. Write a test that exposes a bug in it, if one exists. (Hint: Test the edge cases first (empty input, zero, negative numbers), since slop tends to hide there.)
Sources
Huang et al., A Survey on Hallucination in Large Language Models (ACM TOIS, 2023, rev. 2024) (https://doi.org/10.48550/arXiv.2311.05232)
Kalai et al. (OpenAI), Why Language Models Hallucinate (arXiv, September 2025) (https://doi.org/10.48550/arXiv.2509.04664)
Perry et al. (Stanford), Do Users Write More Insecure Code with AI Assistants? (ACM CCS, 2023) (https://doi.org/10.48550/arXiv.2211.03622)
Spracklen et al., We Have a Package for You! Package hallucinations by code-generating LLMs (USENIX Security, 2025) (https://doi.org/10.48550/arXiv.2406.10279)
Parasuraman and Manzey, Complacency and Bias in Human Use of Automation (Human Factors, 2010) (https://doi.org/10.1177/0018720810376055)
Sharma et al. (Anthropic), Towards Understanding Sycophancy in Language Models (ICLR 2024) (https://doi.org/10.48550/arXiv.2310.13548)
METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (July 2025) (https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)
Shumailov et al., AI models collapse when trained on recursively generated data (Nature, July 2024) (https://doi.org/10.1038/s41586-024-07566-y)
Shanahan, Talking About Large Language Models: why the language of belief and knowledge misleads when applied to a system that predicts text (Communications of the ACM, vol. 67, no. 2, February 2024) (https://doi.org/10.1145/3624724)
10. Debugging and Evaluating AI Systems
Reading generated code, testing, reliability, and benchmarks.
If Chapter 9 says the machine is wrong, this chapter is how you catch it. Students learn to read generated code and write tests, measure reliability, and benchmark outputs. Quality becomes a number, not a vibe.
By the end of this chapter, you will be able to
Read AI-generated code by tracing data flow instead of trusting comments and names
Apply debugging strategies with an AI: compare working vs. broken cases, add logs, and build minimal reproductions
Choose the right evidence for a bug report to an AI: screenshots for visual symptoms, copy-pasted text for errors and logs
Write tests targeted at edge cases rather than just the happy path
Explain the difference between reliability and reproducibility, and how to achieve both
Design a rubric or benchmark to measure AI output quality over time
10.1 Reading code you didn't write
Chapter 9 argued that the machine is confidently wrong. This chapter is how you catch it, and it starts with a skill no test can replace: reading code you didn't author, for the moments that call for it. Most changes you'll verify by running them. When the result is wrong and the agent can't say why, or the code handles money, permissions, or personal data, you read. The mistake beginners make when they do is reading the comments and the variable names (the prose) and feeling reassured. Comments and names describe intent, and the bug is almost always a gap between intent and behavior, so the prose is exactly the part you can't trust.
Read the data flow instead. Pick a value, follow it from where it's created to where it's used, and at each step ask what type and shape it actually has right now, not what it's named, not what the comment promises. This is slower than skimming, and it's the entire game: most AI-introduced bugs are a value that's the wrong shape at one specific step (a string where a number was assumed, a possibly null thing treated as definitely present).
Question assumptions out loud as you go, because the model's mistakes cluster exactly where it quietly assumed the easy case, so ask whether this list ever arrives empty, whether this number can be negative or zero, and what happens the first time this network call fails instead of succeeding. Each question you ask is a place the happy-path-loving model may not have looked, and naming them is how you decide what to test next.
One concrete technique sharpens all of this: before you run the code, predict what it should do on a deliberately tricky input, then run it and compare. When your prediction and the actual result diverge, you've found either a bug or a hole in your own understanding, and both are worth knowing. This 'predict, then run' habit converts reading from passive skimming into an active hunt with a scoreboard.
The debugging loop. It doesn't change much when an AI writes the code; you just run it more often.
Ways to know your code works, from a quick eyeball to watching real users. Higher layers catch what lower ones miss.
10.2 Compare, instrument, isolate
Once you can read the code, you need strategies for when it misbehaves, and the highest-leverage one is differential debugging: comparing a thing that works against a similar thing that doesn't. If the profile page loads but the settings page (built the same way) spins forever, the bug lives in the difference between them, and that difference is a much smaller haystack than either page alone. This comparison is something AI is genuinely good at, so use it deliberately: paste both versions and ask 'this one works and this one doesn't. Investigate what's different and why that difference would cause the failure.' Framing it as a comparison anchors the model in a working reference instead of letting it guess at fixes in a vacuum.
When the difference isn't visible in the code, make the runtime visible: ask the AI to add logging. A prompt like 'add log statements at each step of this request so we can see where the data stops looking right' turns a mystery into a trail of evidence. Run the failing flow, paste the log output back into the conversation, and now the model is reasoning from what actually happened instead of from what the code looks like it should do. This is the single biggest upgrade most beginners can make to their debugging prompts: stop describing the bug from memory and start feeding the AI real output (logs, the exact error text, the actual values).
You often don't even need to add logging. The evidence is already sitting in a console. Every browser ships developer tools (F12 on Windows and Linux, Command plus Option plus I on a Mac, or right-click and choose Inspect in any of them; Safari asks you to enable them once under Settings, then Advanced). The Console tab is where a web page confesses: red error messages, failed network requests, warnings, even when the page itself just looks silently broken, and the same goes for the terminal window running your server or build. Something misbehaves? Open the console, select everything there, and paste the full block into the chat, not a hand-picked fragment. The line that looks like noise to you ('some warning about hydration?') is frequently the exact clue the model needs, and the lines before an error often matter as much as the error itself. Beginners paste one red line and ask 'what's this?' Practitioners paste the whole console and ask 'the page goes blank when I click save. Here's everything the console printed.'
The two strategies combine well. Add the same logging to the working case and the broken case, run both, and diff the logs. The first line where the two trails diverge is very close to the bug, and pasting both trails into the model with 'they diverge at step 3. Why?' is about as well-posed as a debugging question gets. You've converted 'it doesn't work' (which the model can only answer with guesses) into 'here's the exact step where behavior differs' (which it can actually analyze). Rubber-duck debugging, the old practice of explaining your bug aloud to a toy duck until you hear the flaw yourself, survives fully intact. The duck just talks back now.
For anything visual, a screenshot is often the fastest evidence you can hand over. Modern models read images, and a screenshot of the mangled layout, the error dialog, or the blank page communicates in one paste what three paragraphs of 'the sidebar is sort of overlapping the header' can't. Screenshots shine exactly where words fail: spacing and alignment bugs, wrong colors, elements rendering in the wrong place, or 'it looks fine on desktop but broken on mobile' (send both). The working-and-broken comparison applies here too. A screenshot of the good state next to a screenshot of the bad one lets the model spot the visual diff the same way it spots a code diff.
Screenshots have a sharp limitation, though: they capture how things look, not why. A screenshot of an error dialog is worse than the error's actual text, because the model can copy, search, and reason over text precisely, while reading it out of pixels is lossy. A truncated message or a blurry stack trace loses the one detail that mattered. So the rule of thumb is: screenshot the visual symptom, copy-paste the textual evidence (error messages, stack traces, console output, log lines), and send both when you have both. The screenshot says 'here's what I see,' the text says 'here's what the system said,' and a model given both starts from the strongest possible position.
When neither comparison nor logs crack it, isolate: cut the failing behavior out of your app into the smallest program that still misbehaves, which is the classic minimal reproduction [6], and it pays for itself twice over. Half the time, the trimming reveals the bug for you, because the bug was in something you deleted and the problem vanished. The other half, you now have a small, self-contained snippet that fits comfortably in a prompt, so the model sees the whole problem instead of a fragment of a large codebase it has to guess around.
Evidence beats descriptionEvery strategy here is a version of the same move: replace your description of the bug with evidence of the bug. A working/broken comparison, a log trail, and a minimal repro. Each gives the model ground truth to reason from instead of your (possibly wrong) summary of what's happening.
10.3 Testing strategies
Tests are how you make correctness repeatable instead of re-checking by hand every time. They come in a rough hierarchy. Unit tests check one function in isolation. They're fast and precise, and you have lots of them. Integration tests check that pieces work together, like a function and the database it actually writes to. End-to-end (E2E) tests drive a full user flow through the real system, slow but closest to what a user experiences. A healthy project has many unit tests, fewer integration tests, and a handful of E2E tests on the critical paths, a shape usually called the test pyramid [3].
Here's the insight that makes testing AI-generated code pay off: aim your tests at where the model is weak instead of where it's strong. Generated code reliably handles the common, obvious path (that's the most-represented pattern in its training) and reliably under-handles the edges: empty inputs, a single-element list, zero, negative numbers, duplicates, two requests arriving at once. A test suite that just re-walks the happy path mostly confirms what already works. Tests aimed at the edges are where you actually catch bugs.
So write tests adversarially, as if you're trying to break the code, not confirm it. For any function, ask 'what's the weirdest valid input, and what's the input right at the boundary?' and write those cases first. This is also a place to use AI against itself (ask the model to enumerate edge cases for a function, then write tests for them), but verify the cases, since the same blind spots can recur.
A specific habit worth building early: when you find a bug, write a test that reproduces it before you fix it. That test fails, you fix the code, it passes, and now it stands guard forever against that exact bug coming back (a regression). Bugs found in AI-generated code love to reappear when the code is regenerated. A regression test is how you make a fix actually stick.
10.4 Reliability and reproducibility
Reliability is the unglamorous property that a system does the same thing given the same input, every single time. It sounds obvious, but it's exactly what separates a real system from a demo: a demo has to work once, in front of one audience. A reliable system has to work the thousandth time, on input you didn't anticipate, when you're not watching.
AI-assisted development quietly erodes reliability in a specific way. Generated code often leans on sources of non-determinism (the current time, a random value, the order results come back from an external service, shared state between requests) without pinning them down. The result is a flaky test or feature: it passes on your machine this afternoon and fails in CI tonight, and the failure feels random because, underneath, it is.
The cure is to make behavior reproducible by removing the unpinned variables. Fix the seed so 'random' is deterministic in tests. Mock the clock so 'now' is a known value. Isolate test data so one test doesn't depend on another having run first. Control or stub the external calls. Each of these converts 'it worked when I tried it,' the least trustworthy sentence in software, into something you can actually stand behind.
Reproducibility also pays off the moment something breaks. A bug you can reproduce on demand is a bug you can fix. A bug that appears one run in twenty is a nightmare you'll chase for days. So the effort you spend pinning down non-determinism does more than produce clean tests. It's what makes the inevitable debugging session (Chapter 14's incidents, Optional Chapter E's inherited messes) tractable instead of maddening.
10.5 Human oversight
Testing and reproducibility catch a lot, but some decisions should never be fully handed to an automated check at all. They need a human in the loop on principle, not because the tests are weak. The categories are consistent: anything touching money, anything irreversible (deleting data, sending email to real users, publishing publicly), and anything whose consequences land outside your system on actual people.
Notice this is the same reversibility-and-stakes logic as the agent checkpoints in Chapter 8, applied to your development process instead of a running agent. It's one principle wearing two hats: automate freely where a mistake is cheap and undoable, and insist on a human's eyes where it isn't. The cost of a wrong automated decision, not the difficulty of automating it, is what should decide.
Oversight doesn't mean reviewing every line of every change forever. That doesn't scale, and it trains you to rubber-stamp, which is worse than no review because it manufactures false confidence. It means deciding deliberately and in advance which categories of change always get human review before shipping, and then actually honoring that line even when you're moving fast and the diff looks fine.
The deeper point is that 'human oversight' is a design decision you make on purpose, not a vague good intention. Where exactly does a person have to look? What can the system do on its own? Writing that down (for your project, your agents, your deploy process) is what turns oversight from a comforting word into an actual safeguard, and it's a decision Chapter 14 will press on again under higher stakes.
10.6 Benchmarking AI outputs
Tests give a clean pass/fail when there's a single right answer. But much of AI work has no single right answer. Is this summary good, is this reply helpful, and did this prompt change actually improve things? The public coding benchmarks work the same way, whether that's HumanEval scoring generated functions against unit tests that the model never sees in its prompt [5] or SWE-bench asking models to resolve real GitHub issues [4]. Benchmarking is how you bring rigor to those fuzzy questions: you turn 'this feels better' into a number you can compare.
The mechanics are straightforward. Define either a rubric (specific, checkable criteria like 'cites a source,' 'under 100 words,' 'answers the actual question') or a gold-standard set of known-good answers, then score outputs against it and track the score across prompts, models, and time [2]. The discipline this imposes is the real value: it forces you to say precisely what 'good' means, which is often the hardest and most clarifying part.
This is the difference between 'I think my prompt change helped' and knowing it did. Without a benchmark, you're tuning a system on vibes and anecdotes, where a change that fixed the one example you looked at may have quietly broken five you didn't. With one, every change becomes a measurable experiment, and you can improve deliberately instead of wandering. Practitioners call these automated evaluations 'evals,' and serious AI products live or die by them.
At scale, hand-scoring every output gets expensive, so a common technique is LLM-as-judge: use a model to grade another model's output against your rubric [1], which is powerful and carries an obvious risk. A judge that shares the generator's blind spots, or that can be flattered, isn't a neutral referee. So you validate the judge against human scores on a sample before trusting it, which is just this whole chapter's lesson applied one level up: even your verification tools need verifying.
Summary
When you didn't write the code, your job shifts from authoring to judging. We covered reading generated code, then the core debugging strategies: differential debugging (compare a working case against a similar broken one, since the bug lives in the difference), asking the AI to add logs so it reasons from real runtime output instead of guesses, screenshots for visual symptoms paired with copy-pasted error text, and minimal reproductions when all else fails. We covered writing tests aimed at edge cases, since the happy path already works, reproducibility, human oversight, and benchmarking, including using a model as a judge once you've validated it against humans. A passed test is necessary but not sufficient, since tests only check what someone thought to test.
Key terms
Differential debugging. Finding a bug by comparing a working case against a similar broken one, since the bug lives in the difference between them.
Logging. Emitting messages about what a program is doing at runtime so you can see where behavior diverges from expectation instead of guessing.
Console. The browser dev-tools tab (or server terminal) where errors, warnings, and log output appear. Copy-pasting its full contents into an AI chat is one of the highest-value debugging moves.
Minimal reproduction. The smallest self-contained program that still exhibits a bug, small enough to reason about (or paste into a prompt) whole.
Unit test. A test that checks one function or unit in isolation.
Integration test. A test that checks multiple pieces working together, such as a function plus the database it writes to.
End-to-end (E2E) test. A test that exercises a full user flow through the real system, front to back.
Edge case. An input at the boundary of expected behavior (empty, zero, negative, duplicate, very large), where AI-generated code most often breaks.
Reliability. The property that a system behaves the same way given the same input, every time.
Reproducibility. Being able to get the same result again by pinning sources of non-determinism (random seeds, time, external state).
Flaky test. A test that passes or fails inconsistently across runs, usually due to unpinned non-determinism.
Benchmark. A defined task set used to score and compare outputs, turning 'feels better' into a measurable number.
Eval. An automated evaluation that scores model or system output against expected results or a rubric, run repeatedly as the system changes.
LLM-as-judge. Using a model to grade another model's output against a rubric, scalable but itself needing validation against human judgment.
Rubric. Specific, checkable criteria (e.g. 'cites a source,' 'under 100 words,' 'answers the actual question') used to score AI output consistently, whether by a human or an LLM-as-judge.
Regression. A previously working behavior that breaks after a change. Regression tests guard against reintroducing old bugs.
Human oversight. A deliberate, decided-in-advance rule that certain categories of change (anything touching money, anything irreversible, anything affecting real people) always get a human's review before shipping, regardless of what automated checks say.
Check your understanding
Q1. What kind of bugs does AI-generated code most often leave behind?
Edge cases like empty input, duplicates, or concurrent requests
Bugs in the most common, happy-path case users hit first
None, since generated code is checked before it's emitted
Syntax errors that the parser catches
Answer: A. Models tend to nail the common case. Tests aimed at edges (empty, negative, duplicate, concurrent) catch what the happy path won't.
Q2. Feature A works, but the nearly identical feature B fails. What's the most effective way to ask an AI for help?
Paste only feature B and ask 'why is this broken?'
Paste both and ask what differs and why that difference breaks it
Ask it to rewrite feature B from scratch, showing it neither version
Describe both features from memory without pasting any code
Answer: B. Differential debugging anchors the model in a working reference. The bug lives in the difference between the two, which is a far smaller search space than either feature alone.
Q3. You hit an error dialog with a stack trace. What's the best evidence to give the AI?
A screenshot of the dialog, since images are richer than text
A from-memory summary like 'it said something about null'
The copied error text and stack trace, plus a screenshot if visual
Nothing at all, since the AI should find it from the code alone
Answer: C. Models reason over exact text precisely, while reading text out of pixels is lossy. Screenshot the visual symptom, copy-paste the textual evidence, and send both when you have both.
Q4. What does it mean for a test to be 'flaky'?
It fails on every single run without any exception
It exercises only the interface components and nothing else
Flakiness is generally a sign of unusually well-covered code
It passes or fails inconsistently because of timing or randomness
Answer: D. Flakiness usually comes from unpinned non-determinism. Fixing seeds, mocking time, and isolating test data restores reproducibility.
Q5. What does a benchmark let you do that a gut feeling can't?
Turn quality into a comparable number instead of an impression
Nothing, gut feelings and benchmarks are equally rigorous
Eliminate the need for any human review
Guarantee the AI never makes mistakes
Answer: A. A rubric or gold standard makes 'did this change actually help' answerable with evidence instead of vibes.
Practice
Exercise 1. Take an AI-generated function and write three unit tests aimed specifically at edge cases (not the happy path). Run them. Did any fail? (Hint: Try empty input, the largest/smallest plausible value, and an unexpected type if the language allows it.)
Exercise 2. Take a bug you (or someone) hit recently where one thing worked and a similar thing didn't. Ask an AI to add logging to both paths, run them, and paste both log trails back with 'where do these diverge and why?' Did the divergence point match where the bug actually was? (Hint: If you don't have a live bug handy, break something on purpose: duplicate a working function, change one subtle thing (an off-by-one, a swapped argument), and practice the compare-and-instrument loop on it.)
Exercise 3. Find a visual bug (or create one by mangling some CSS). File two bug reports to an AI: one describing the problem in words only, one with a screenshot plus the relevant copy-pasted code. Compare the quality of the two responses. (Hint: Note what the words-only report got wrong or vague. That gap is exactly what the screenshot carries for free.)
Exercise 4. Design a 5-point rubric for grading the quality of an AI-generated email reply (e.g. tone, accuracy, completeness, length, actionability). Score two sample replies against it. (Hint: Make each rubric point specific enough that two different people would give the same score. Vague criteria defeat the purpose of a rubric.)
Sources
Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023) (https://doi.org/10.48550/arXiv.2306.05685)
Anthropic, Define success criteria and build evaluations (Claude docs) (https://platform.claude.com/docs/en/docs/test-and-evaluate/develop-tests)
Vocke, The Practical Test Pyramid (martinfowler.com, February 2018) (https://martinfowler.com/articles/practical-test-pyramid.html)
Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (ICLR 2024) (https://doi.org/10.48550/arXiv.2310.06770)
Chen et al. (OpenAI), Evaluating Large Language Models Trained on Code: the HumanEval benchmark (arXiv, 2021) (https://doi.org/10.48550/arXiv.2107.03374)
Stack Overflow, How to create a minimal, reproducible example (https://stackoverflow.com/help/minimal-reproducible-example)
11. Multimodal AI
Vision, speech, image, and video models in real apps.
Text is just one modality. This chapter expands the toolkit to vision, speech, image generation, and video. It shows how to combine these modalities into a single application.
By the end of this chapter, you will be able to
Call a vision-capable model to extract information from an image or document
Describe how speech-to-text and text-to-speech combine with a text model in a voice pipeline
Explain controllability trade-offs in image and video generation
Apply responsible-disclosure judgment to AI-generated synthetic media
Predict what a coding agent can ingest directly (links, screenshots, video) and when it's decomposing the input behind the scenes
11.1 Vision models
Text is just one modality (one type of data a model can work with), and the most useful applications increasingly combine several. A vision-capable model accepts an image alongside text and can describe what's in it, answer questions about it, read the text inside it (OCR, effectively for free), or pull structured data out of a document, receipt, or screenshot. The model genuinely 'sees' the image in the sense that it can reason about its contents, not just label it.
The reassuring news for a builder is that almost nothing new is required to use this. Calling a vision model looks like the model calls from Chapter 7 with an image attached as another input, so the same authentication, error handling, and cost concerns apply. Multimodality expands what the model can take in, not how you talk to it, which is why the integration skills you already have transfer directly.
This unlocks a surprisingly broad class of features with very little code. Consider 'extract the line items from this uploaded invoice,' 'describe this photo for a visually impaired user,' or 'tell me which of these screenshots shows an error.' Tasks that a few years ago needed specialized computer-vision pipelines are now a single API call. Recognizing when a problem is secretly a vision problem is half the skill.
The skepticism from Chapters 9 and 10 carries over intact, though. A vision model can misread a blurry number, miss something in a cluttered image, or confidently describe a detail that isn't there, which is hallucination in pixels instead of text. For anything that matters, like reading a dollar amount off a receipt, you verify the extraction the same way you'd verify any other model output.
One model, many input types. Multimodal models blur the line between 'read this image' and 'reason about it.'
Upload image โ Model reads โ Reasoning โ Text / code out
How a vision model turns a screenshot or photo into text or code you can act on.
11.2 Speech models
Speech adds the ear and the voice. Speech-to-text (STT, also called automatic speech recognition) transcribes spoken audio into written text. Text-to-speech (TTS) goes the other way, synthesizing natural-sounding speech from a script. Modern versions of both are strikingly good. TTS voices are now hard to distinguish from human recordings, which is part of why this chapter ends on ethics.
The architectural move that matters is keeping the modalities separate and letting a text model do the thinking in between, since a voice assistant isn't one magic model. It's a pipeline: STT turns the user's speech into text, a regular language model reasons about that text (possibly using tools and RAG from Chapter 4), and TTS speaks the answer back, giving you voice in and voice out with familiar text-based reasoning in the middle.
Seeing it as a pipeline is liberating, because it means everything you already know plugs straight in. The 'brain' of a voice app is just the text model and tools you've been building all along. STT and TTS are adapters on the front and back. This same separation-of-concerns shape is exactly the vision pipeline from the last section, with audio swapped in for images.
And the same caution returns at the seams. STT is imperfect (accents, background noise, and homophones produce transcription errors), and every error it makes becomes wrong input to the reasoning step, which then confidently reasons about something the user never said. A well-built voice app accounts for the transcription being slightly wrong (confirmation, tolerance for near-matches) instead of assuming the text it got is exactly what was spoken.
11.3 Image generation
Generation flips the direction: instead of reading an image, the model creates one from a text prompt or edits an existing image given instructions. Many of these systems are diffusion models, which work by starting from random noise and iteratively refining it toward something that matches the prompt. Others generate an image the way a language model generates text, predicting it piece by piece inside the same model that read your prompt. The mechanism is worth knowing either way, because it explains both the dreamlike strengths of these models and their characteristic weirdness with fine detail.
The central practical challenge is controllability: how reliably the output matches the specific thing in your head. It varies enormously by model and by how precise your prompt is. A vague prompt yields something plausible but probably not what you pictured. A detailed one (composition, lighting, style, what to exclude) narrows the result. Getting consistent output means iterating on the prompt, exactly the refinement loop from Chapter 2, now with a visual result you judge instead of code you test.
It's worth being honest about the limits, because they're where 'looks right' bites in image form. Text inside images has improved fast, and the frontier models now handle a short string of words well. A full paragraph still breaks down after a few hundred characters, and models below that tier still garble even short strings. Exact counts of objects and precise spatial relationships ('the cup to the left of the plate') stay shaky, because these models are matching overall statistical plausibility, with no rule being enforced anywhere. The output can look fantastic and still get the one detail you cared about wrong, so you check generated images against your actual requirement, not just your aesthetic reaction.
Used well, image generation is a genuine superpower for a small team: placeholder art, marketing visuals, icons, mockups, concept exploration, all in seconds. The judgment it demands is partly taste (is this actually good) and partly the ethics the next section raises, because the same ease that makes it useful also makes it easy to misuse.
11.4 Video generation and synthetic media ethics
Video generation extends image generation across time, and it's the current frontier: more expensive, less controllable, and improving alarmingly fast, which makes the caveats from image generation bite harder still. Temporal consistency, keeping a face or object stable frame to frame, is genuinely hard. But the trajectory is clear enough that the interesting questions are quickly becoming less 'can it' and more 'should it.'
That's because generated images, audio, and especially video are now realistic enough to be difficult to distinguish from real recordings. A deepfake, synthetic media that convincingly depicts a real, identifiable person doing or saying something they never did, is no longer exotic. Once the technical barrier falls, the meaningful constraint is human judgment, not capability, and that's a different kind of question than the rest of this book has asked.
The responsible-use bar is whether you should make this, and whether the audience knows it's synthetic. Disclosure is the floor, which means labeling AI-generated media, especially anything depicting real people or presented as a real event. Disclosure is becoming a legal requirement as well as an ethical one, because the European Union's AI Act transparency rules began applying on August 2, 2026, requiring anyone deploying a system that produces a deepfake to disclose that the content was artificially generated, and requiring providers of generative systems to mark their output in a machine-readable way [8]. There's growing technical support for it through provenance metadata and watermarking that mark or trace a file's synthetic origin, and the industry standard for attaching that history to a file is the C2PA Content Credentials specification [9], though these help only partially, leaving the actual safeguard as the choice you make before you publish.
This is the chapter where 'just because you can build it doesn't mean you should' stops being a slogan and becomes a concrete decision you'll personally face the first time you can generate a convincing fake of a real person. Optional Chapter B and Chapter 14 develop the societal and legal dimensions. The personal one is simpler and lands first: the capability is in your hands now, and what you do with it is on you.
11.5 Combining modalities
The real power shows up when you chain modalities into one experience. Picture an app where a user speaks a question about a photo they've uploaded: STT transcribes the speech, a vision-capable text model reasons over both the transcript and the image, and TTS reads the answer aloud. That's four modalities in one flow, and none of the individual pieces is anything you haven't now seen.
What makes these apps feel magical is also what makes them fragile: they're pipelines, and a pipeline is only as reliable as its shakiest stage. An error early (a misheard word, a misread image) doesn't stay contained. It propagates, and every downstream stage confidently builds on the bad input. The system can fail in a way no single component would, because the failure is in the handoffs.
So a multimodal pipeline demands the per-stage skepticism of Chapters 9 and 10, applied at every seam, not only at the final output. Ask, stage by stage: what fails here, and what does the next stage do with that failure? Design for a slightly wrong transcription or an incomplete image read. Confirmations, fallbacks, and tolerances are what separate a durable multimodal app from an impressive demo that breaks on the second real user.
Step back, and the lesson is encouraging, because multimodality is composition and not a new discipline. Vision, speech, generation, and text models are building blocks with familiar interfaces, and combining them is mostly about wiring and verifying the seams. The genuinely new thing you bring is product imagination, seeing which combination actually solves a real problem, which is exactly what Chapter 12 turns to next.
11.6 Feeding the real world to your coding agent
Everything so far treated modalities as things you wire into an app you're building. Your coding agent is a multimodal consumer too, and knowing what it can actually ingest changes how you work with it every day, starting with links. Paste a URL into Claude Code, and it will fetch the page for you: a web-fetch tool downloads it, converts the HTML to markdown, and runs your question against that text with a small, fast model, so what reaches the agent is usually that model's answer and not the page itself. It asks permission the first time it reaches a domain outside a built-in preapproved documentation list [1], while Codex is choosier. As of mid-2026, its CLI answers web questions from a cached index by default and fetches live pages only when you opt in with a flag [3], and its cloud sandboxes keep the internet off during agent runs unless you enable it per environment. The caution is deliberate. A page your agent reads is also a page that can carry instructions planted for your agent, which is Chapter 14's prompt injection arriving through a docs link.
The detail worth internalizing is that fetching is a text operation. The agent works from the page's converted text, not the rendered pixels, so it can quote your documentation accurately and still have no idea that the header is overlapping the logo. Seeing takes a browser (or, for anything beyond the browser, the computer-use agents of Section 8.7, which operate whole applications through screenshots). Claude Code drives a browser through the Claude in Chrome extension, which can open a page, click, fill forms, take screenshots, and record a short GIF of an interaction [2], and Codex ships a similar built-in browser in ChatGPT on the web and in the desktop app that opens a local page, reproduces a layout bug, and screenshots the result, though as of mid-2026 that browser isn't available from the Codex CLI or the Codex IDE extension [4]. Both tools also take images you hand them directly: Claude Code reads PNG and JPG files from disk and accepts pasted screenshots [1], and codex -i screenshot.png does the same from a flag [5]. Chapter 10 told you to paste screenshots as debugging evidence, and this is why that works.
Video is where the tools genuinely part ways, so the claim has to be made carefully. As of mid-2026, neither the Claude API nor OpenAI's models accept a video file as input, and OpenAI's transcription endpoint will take an mp4 but hears only the audio track [7]. Gemini reads the whole thing natively: hand it an mp4, and it processes both streams, sampling the visuals at about one frame per second and the audio alongside, with room for up to an hour of footage at default resolution on a 1M-context model, and public YouTube links work in preview [6]. Hand the same mp4 to Claude Code, and something more interesting happens. A capable agent decomposes it, running ffmpeg to pull frames it can read as images, extracting the audio, and sending that to a transcription model before reasoning over the pieces. That's the pipeline thinking of 11.5 performed by the agent on its own input. It's a workaround the agent improvises, so expect to nudge it, and expect it to miss things a native reader would catch between the sampled frames.
You hand it...
Claude Code
Codex
Gemini API
A web link
Fetches it, converts the HTML to markdown, permission-gated by domain
Cached search index by default; live fetching and cloud network access are opt-in
Reads public YouTube URLs natively (preview)
A screenshot or image file
Reads PNG/JPG from disk, accepts pasted screenshots
codex -i file.png, or paste into the composer
Images natively
A page to look at, rendered
Claude in Chrome extension: open, click, screenshot, record GIFs
Built-in browser in ChatGPT on the web and in the desktop app; not in the CLI or IDE extension
No browser of its own; you send it what to look at
An mp4 with audio
No native video; decomposes via ffmpeg frames plus a transcription step
No native video; the transcription endpoint hears the audio only
Native: both streams, sampled at about 1 frame/second, up to an hour at default resolution
Ask what it actually receivedWhen an agent's answer about a page or file seems off, ask it what it actually ingested. 'I fetched the page as markdown' and 'I looked at a screenshot' are different evidence, and the follow-up takes ten seconds. The gap between fetching and seeing explains a whole family of confusing agent mistakes.
Summary
Models now work across text, images, audio, and video, and the API patterns barely change. We built and verified each modality conversion in isolation before chaining them. Multimodal pipelines tend to show their seams on tricky inputs. Hallucination in pixels reads exactly like hallucination in text, so the same Chapter 9 skepticism applies. Disclosure matters when output is synthetic. The closing section turns the lens on your own tools: coding agents fetch links as markdown, not pixels, need a browser to actually see a page, and handle video either natively (Gemini) or by decomposing it into frames and a transcript (Claude Code, Codex).
Key terms
Modality. A type of data a model works with, such as text, image, audio, or video.
Multimodal model. A model that accepts or produces more than one modality, e.g. reading an image and answering in text.
Vision model. A model that accepts images and can describe them, answer questions about them, or extract data from documents and screenshots.
OCR (Optical Character Recognition). Extracting text from an image or scanned document. Modern vision models do this implicitly.
Speech-to-text (STT). Transcribing spoken audio into written text, also called automatic speech recognition (ASR).
Text-to-speech (TTS). Synthesizing natural-sounding spoken audio from written text.
Image generation. Producing a new image from a text prompt or editing an existing one from instructions.
Diffusion model. A common architecture behind image and video generation that produces output by iteratively denoising random noise toward a prompt.
Video generation. Generating moving footage from a prompt, extending image generation across time. More expensive and less controllable than image generation, with temporal consistency (keeping a face or object stable frame to frame) as its hardest technical problem.
Deepfake. Synthetic media that realistically depicts a real, identifiable person doing or saying something they did not.
Provenance / watermarking. Techniques and metadata for marking or tracing whether media is AI-generated, supporting responsible disclosure.
Web fetch. An agent tool that downloads a page and converts its HTML to markdown for the model to read. The agent gets the text, not the rendered layout.
Fetching vs. seeing. The distinction between reading a page's converted text and looking at its rendered pixels through a browser or screenshot. An agent that fetched a page can quote it accurately and still miss a visual bug.
Video decomposition. The workaround an agent uses on video that its model can't ingest natively: extract frames as images (ffmpeg), extract and transcribe the audio, then reason over the pieces.
Check your understanding
Q1. How does calling a vision-capable model differ from a normal text-only model call?
It requires an entirely different API and a separate provider account
You attach an image alongside the text, and the call pattern is the same
Vision models accept images only and reject any accompanying text
It requires no prompt text at all, only the image file
Answer: B. Vision support extends the same request pattern to accept an image input. It's an additional input type, not a different paradigm.
Q2. What's the core ethical question this chapter raises about synthetic media?
Whether it's technically possible to generate, which it usually is
Whether the output file format is supported by common players
Whether you should make it, and whether the audience knows it's synthetic
Synthetic media raises no ethical questions that are worth weighing
Answer: C. The bar is disclosure and intent, not capability, especially when depicting a real, identifiable person.
Q3. In a chained multimodal pipeline (speech in, text reasoning, speech out), what should you be skeptical of?
Nothing, since chaining modalities cancels errors out along the way
Only the final output, since intermediate steps are checked for you
Multimodal pipelines avoid the problems catalogued in Chapter 9
Every conversion step, since a bad transcription poisons the reasoning
Answer: D. Every modality conversion is a place errors can creep in. The same skepticism that applies to text generation applies to each stage here.
Q4. You paste a URL into Claude Code and ask whether the page's layout looks broken. It fetches the page and says everything looks fine, but your browser shows the header overlapping the logo. What most likely happened?
It read the HTML as markdown, which carries no layout, so it never saw it
The page changed in between your look and the agent's own fetch
The agent's vision model misread the screenshot it captured
Web fetching was blocked, so the agent simply guessed an answer
Answer: A. Fetching is a text operation. The agent got the page's content as text and reasoned about that, so it can be right about every word and blind to every pixel. Checking rendering takes a browser tool or a screenshot you hand it.
Q5. You drop an mp4 with narration into your project folder and ask a coding agent to summarize it. Which description of what happens next is accurate as of mid-2026?
Every major model reads mp4 natively, so all the tools behave alike
Gemini reads video natively, while Claude Code splits it into frames and audio
No model handles video at all, so the request fails in every tool
Only the audio track ever matters, since models can't process frames
Answer: B. Gemini processes both the visual and audio streams natively, sampling around one frame per second. The Claude and OpenAI APIs don't accept video, so an agent like Claude Code handles an mp4 by decomposing it, which works but is a workaround with real gaps between sampled frames.
Practice
Exercise 1. Build a small app that accepts an uploaded photo and makes a vision model call to answer one specific question about it (e.g. 'what's the dominant color of this image?'). Verify the answer yourself. (Hint: Pick a question with a clearly checkable right answer so you can actually tell whether the vision call got it right.)
Exercise 2. Generate the same image prompt three times with an image generation model. Note what's consistent and what varies, and rewrite the prompt to be more specific based on what you saw. (Hint: Add concrete details (composition, lighting, style) one at a time and observe which addition actually narrows the variation.)
Exercise 3. Run the fetching-versus-seeing experiment on a page you know has a visual quirk. First give your coding agent the URL and ask about the layout, then give it a screenshot of the same page and ask again. Write three sentences on what each version could and couldn't tell you. (Hint: Ask the agent what it actually ingested in each case. The first answer comes from converted markdown, the second from pixels, and the difference in what it notices is the whole lesson.)
Sources
Claude Code tools reference: WebFetch, WebSearch, and Read tool behavior (https://code.claude.com/docs/en/tools-reference)
Claude in Chrome: browser automation from Claude Code (https://code.claude.com/docs/en/chrome)
Codex web search: cached vs. live fetching (https://learn.chatgpt.com/codex/web-search)
Codex browser (ChatGPT on the web and the desktop app; not in the CLI or IDE extension) (https://learn.chatgpt.com/codex/browser)
Codex image inputs: codex -i and pasting (https://learn.chatgpt.com/codex/image-inputs)
Gemini API video understanding: formats, frame sampling, audio track, limits (https://ai.google.dev/gemini-api/docs/video-understanding)
OpenAI speech-to-text: supported transcription input formats, including mp4 (https://developers.openai.com/api/docs/guides/speech-to-text)
European Commission, transparency obligations under Article 50 of the AI Act, applying from August 2, 2026 (https://digital-strategy.ec.europa.eu/en/faqs/transparency-obligations-under-article-50-ai-act)
C2PA, Coalition for Content Provenance and Authenticity: the Content Credentials standard (https://c2pa.org/)
12. AI Product Design
User research, MVPs, and choosing problems worth solving.
Being able to build anything makes choosing what to build the hard part. This chapter covers user research, product-market fit, MVP scoping, and human-AI interaction design. Students learn to aim their new powers at problems that matter.
By the end of this chapter, you will be able to
Evaluate whether a problem is worth building for using the worthwhile-problem criteria
Conduct a user interview that surfaces real evidence of pain rather than polite enthusiasm
Scope an MVP by cutting non-essential features and stating explicit non-goals
Apply the Feature Graveyard test to assess an AI product idea's defensibility
12.1 Choosing a worthwhile problem
Here's the central irony of this whole book. Now that you can build almost anything, the hard part is no longer building. It's choosing what to build. When the cost of making something drops toward zero, judgment becomes the binding constraint. The most common failure is a perfectly built solution to a problem nobody actually had.
That failure even has a name: a solution in search of a problem. It happens when you start from a cool technology or a clever idea and reverse-engineer a justification for it, instead of starting from a pain someone genuinely feels. The technology is so fun to build with that it's easy to skip the question of whether anyone needs the result, and an impressive thing nobody wants is still nobody's want.
There's a reason this trips up strong technical people in particular. As Paul Graham observes in his essay 'Why Smart People Have Bad Ideas,' most of us spend fifteen to twenty years being handed problems to solve [1]. Every exam, assignment, and interview question arrives already chosen, because grading only works when everyone solves the same one. You get very good at solving and almost no practice at choosing, so when the problem is finally yours to pick, you grab the first idea you had or the technology you find most fun to build with. The fix is a discipline, not a gift, and it compresses to four words that became a startup mantra: make something people want. Choosing is learnable, and this chapter is where you start practicing it.
Worthwhile problems share three traits worth checking explicitly. First, someone feels real, recurring pain. Recurring matters. A one-time annoyance rarely motivates anyone to adopt a new tool. Second, that someone is reachable: you can actually find and talk to them, which you'll need to do constantly. Third, the pain is specific enough that you can state in one sentence what 'fixed' looks like. Vague pain means you won't know if you've succeeded.
Run a candidate idea through those three before writing a line of code. If you can't name who hurts, can't reach them, or can't define 'fixed,' you're probably about to build a solution in search of a problem, and the cheapest moment to discover that is now, on paper, not after weeks of building. The rest of this chapter is essentially tools for pressure-testing exactly these three things.
One more habit makes those checks pay off, which is having more than one candidate to run them against. The instinct of most first-time builders is to fall in love with a single idea and defend it, when the better instinct is to generate many quickly and hold each of them loosely. Ideas are cheap and validated ones are rare, so treat your first one as a hypothesis instead of a destination [8].
Discover โ Define โ Develop โ Deliver
The classic design path: diverge to explore, converge to decide, twice [3]. AI speeds up the middle, not the judgment.
Build โ Measure โ Learn โ (repeat)
The product loop. Getting something in front of users early is what tells you whether you're building the right thing.
12.2 User research
User research is the antidote to building in a vacuum: you talk to the people who'd actually use the thing, before you build much of it. It feels slow and unglamorous next to building, which is exactly why it's skipped and exactly why skipping it is the most expensive mistake in the chapter, since a week of conversations can save months building the wrong product.
The technique that separates useful research from flattering noise is in how you ask. Don't pitch your idea and fish for approval. People are nice, and 'that sounds cool' costs them nothing and tells you nothing. Instead, ask open questions about how they handle the problem today ('walk me through the last time this came up') and listen for the texture of real life, not their reaction to your solution. Rob Fitzpatrick's The Mom Test is the standard short treatment of how to ask so that even people who like you can't mislead you [4].
What you're listening for specifically is a workaround: a hacky spreadsheet, a manual routine, a tool bent to a purpose it wasn't built for. A workaround is hard evidence of real, recurring pain, because the person already spent effort to relieve it, voting with their time instead of just their words. Enthusiasm for your pitch is only a wish, while an existing workaround is a fact.
Be alert to the difference between what people say and what they do, because they diverge constantly and good research weights the doing. Someone can sincerely say they'd pay for a feature and never actually use it. Someone else can shrug about your idea while clearly drowning in the exact problem it solves. Behavior (current workarounds, time spent, money already spent) is the signal. Stated preference is decoration on top.
12.3 MVP design
An MVP (minimum viable product) is the smallest version of an idea that still delivers real value to a real user, where every word in that phrase is doing work. The term comes from the lean-startup method, where the point of a first version is to start the build-measure-learn loop and produce real learning as fast as possible [2]. The most-abused word is 'viable.' Viable means someone would actually use it and benefit. A slick demo that helps no one fails that test. A rough tool that solves the core problem passes it, even if investors yawn.
Scoping an MVP is mostly an exercise in ruthless subtraction. Start from the one piece of value the product must deliver, then cut everything that doesn't serve that, including features that feel important and that you'd genuinely like to build. The discipline is hard precisely because the cut features aren't bad ideas. They're just not necessary to learn whether the core idea works, which is all version one is for.
This is where Chapter 2's non-goals earn their keep at the product level. Writing down what you're deliberately not building in v1 ('no teams, no mobile app, no integrations yet') protects you from your own scope creep and from the AI's eager helpfulness, since a model asked to build a todo app will cheerfully add collaboration, tags, and reminders you never scoped. Explicit non-goals keep both of you honest.
The reason to stay small is speed of learning, not laziness. The entire point of an MVP is to get something real in front of real users quickly so you can find out whether you're right, and every extra feature delays that answer while increasing what you've sunk into a direction you haven't validated. Ship the smallest honest version, learn, then decide what to build next from evidence instead of guesses.
12.4 Product-market fit
Product-market fit (PMF) is the moment a product stops feeling like a boulder you're pushing uphill and starts feeling like it's being pulled out of your hands [5]. Enough of the right people want it that growth gets easier instead of harder. It's the thing every startup is really chasing, and most of the work before it is just searching for it.
The defining trait of PMF is that you read it from behavior, not from opinions, least of all your own. Founders are the worst judges of their own product's fit because they're emotionally invested and see every feature's potential. The honest signals live in what users do: do they come back without being nudged (retention), do they tell other people (referrals), and do they get upset if you take it away? Those are facts, where 'I love it' is a pleasantry.
Retention is usually the sharpest single signal, sharper than sign-ups. It's easy to get people to try something once (a good launch, a clever post), and that initial spike flatters you into thinking you've made it. Whether they're still using it in three weeks is the question that actually matters, because real, recurring value is the only thing that brings people back on their own.
This is the through-line that ties the chapter together: you find PMF by shipping a small thing to real users and watching what they actually do, then iterating toward the version they can't stop using, not by polishing privately until it feels perfect to you, because building has become cheap while reading the signal honestly has not. That reading, and the steering that follows from it, is judgment no model can do for you.
12.5 Human-AI interaction design
When the product itself contains an AI feature, the seam where the user meets that AI becomes its own design problem, and a distinctively hard one, because the AI is powerful, useful, and (per Chapter 9) sometimes confidently wrong. Three questions frame the design: how much control does the user keep, how transparent is the system about what it's doing and how sure it is, and what happens when it's wrong?
The mindset that separates good AI UX from bad is treating failure as expected, not exceptional. A confident interface that's silently wrong some of the time erodes trust catastrophically the first time a user gets burned, because they had no warning and no recourse. So design for the wrong answer up front: make AI actions reviewable and reversible so a mistake is a shrug and an undo rather than a disaster.
Two specific practices do most of the work. Graceful failure means the user can always notice, correct, or undo what the AI did: drafts they approve rather than actions taken behind their back, an easy edit, a clear undo. Explainability means surfacing why the system produced an output and how confident it is, so the user can calibrate. An honest 'I'm not sure about this' beats false confidence every time, because it tells the user exactly when to look closely.
The goal of all of it is trust calibration: designing so users trust the AI exactly as much as it has earned, neither blindly (which the model's fluency invites) nor not at all (which wastes the feature). That's the same calibrated-trust idea from Chapter 9, now turned outward: Chapters 9 and 10 taught you to calibrate your own trust in AI. Here you're designing the interface that helps your users calibrate theirs.
12.6 The Feature Graveyard
There's a hazard unique to building on top of AI that you have to factor into what you choose to build: the platform you depend on may simply absorb your product. A huge share of early AI startups were, in effect, a clever prompt and a nice interface wrapped around a model API, like 'a tool that summarizes PDFs' or 'a tool that writes better emails.' Then the model providers themselves (Anthropic, OpenAI, Google) shipped that same capability as a built-in feature, for free, to users who were already there. The Feature Graveyard is where those startups go, killed by the platform underneath them growing upward into their territory.
This isn't new. Apple did it to Mac apps so often it got a name, 'getting Sherlocked,' after Sherlock, an Apple search tool whose third version took on the features of a third-party app called Watson [7]. What's new is the speed and the scale of the damage. A frontier lab can add a feature in a release note and vaporize an entire category of startups overnight, because everyone building thin wrappers around the same API was one model update away from redundant the whole time. Investors call the danger zone around a dominant platform the 'kill zone' for exactly this reason, and economists have studied the way venture funding dries up for startups that land inside it [6].
The lesson isn't 'don't build on AI'. Almost everything in this book is built on AI. The lesson is to ask, about any idea: if the model provider shipped this themselves next quarter, would I still have a business? If the honest answer is no, you're building a feature, not a product, and you're standing in the graveyard's waiting room. Defensibility has to come from something the platform won't or can't easily replicate: deep integration into a specific workflow, proprietary data, a regulated niche, a community, a brand, or genuine domain expertise the general-purpose lab has no reason to chase.
Practically, treat 'thin wrapper around a single API' as a red flag during the worthwhile-problem check from 12.1, not a green light just because it's easy to build. The easiest AI products to build are often the easiest for the platform to eat, which is a deeply uncomfortable correlation. Aim instead for the unglamorous, deeply embedded problems that a lab optimizing for a billion general users will never bother to solve, which is where the ground is solid.
Summary
Building is now the easy part. The hard part is choosing what's worth building. We covered finding a worthwhile problem through user research, scoping an MVP with explicit non-goals, and designing for the UX of being wrong. The Feature Graveyard is full of ideas that demo well but no one uses. Behavior is the signal worth trusting, and an existing workaround is the strongest evidence of real pain.
Key terms
User research. Talking to and observing real potential users to find genuine pain before building much.
Worthwhile problem. One with real, recurring pain, reachable users, and a one-sentence definition of what 'fixed' looks like.
MVP (Minimum Viable Product). The smallest version that delivers real value to a real user, not the smallest impressive demo.
Scope. The set of things an MVP does and, just as importantly, explicitly does not do yet. Scoping is an exercise in ruthless subtraction: cut everything that doesn't serve the one piece of value the product must deliver, and write the cuts down as non-goals.
Product-market fit. The point where enough of the right users want what you built that growth feels pulled, not pushed.
Retention. The share of users who come back over time, usually a stronger signal of value than sign-ups.
Human-AI interaction design. Designing the seam where users meet an AI feature: control, transparency, and what happens when it's wrong.
Trust calibration. Designing so users trust the AI exactly as much as it deserves, neither blindly nor not at all.
Graceful failure. Designing AI features so that when they're wrong, the user can easily notice, correct, or undo the result.
Explainability. Surfacing why an AI system produced a given output and how confident it is, so users can judge it.
Feedback loop. Shipping a small thing to real users, watching what they actually do, and iterating toward the version they can't stop using, instead of polishing privately until it feels perfect. The loop itself is the product strategy, not any single launch.
Wizard of Oz prototype. Faking an AI feature with a human behind the scenes to test demand before building the real thing.
The Feature Graveyard. The fate of startups whose AI platform (Anthropic, OpenAI, Google) ships their core feature as a built-in, eliminating the whole product category overnight.
Sherlocking. A platform absorbing a third-party product's functionality into its own offering, named for Sherlock, an Apple tool whose third version took on the features of a third-party app called Watson, which is the mechanism behind the Feature Graveyard.
Thin wrapper. A product that's mostly a prompt and UI around a single model API, with little defensible value of its own, the most likely candidate for the Feature Graveyard.
Defensibility. Something a product has that the underlying platform won't easily replicate: proprietary data, deep workflow integration, a regulated niche, a community, or real domain expertise.
Check your understanding
Q1. What's the strongest signal of real user pain during an early interview?
They say your pitch sounds genuinely cool and promising
They listen politely and ask no questions at all
They describe a workaround they already built for the problem
They agree with every single thing you propose to them
Answer: C. An existing workaround is behavioral evidence of real, recurring pain. Enthusiasm for a pitch is much weaker evidence.
Q2. What should an MVP optimize for?
Being as impressive as it possibly can be in a live demo
Including every feature you and your team can think of
Matching every feature your closest competitor already ships
Real value to a real user, at the smallest possible scope
Answer: D. An MVP is the smallest thing that's actually useful, not the smallest thing that looks good in a pitch.
Q3. How is product-market fit best measured?
By usage behavior: retention, return visits, word of mouth
By how proud the team feels about what they built
By the total number of features shipped so far
By how polished and finished the interface looks
Answer: A. Fit shows up in what users actually do (come back, refer others), not in internal impressions of quality.
Q4. Your startup idea is 'an app that uses a model API to summarize PDFs.' What's the biggest strategic risk?
The model isn't yet capable enough to summarize PDFs well
The platform underneath you could ship it as a built-in feature
PDFs will fall out of use as a document format
There's little risk, since it's easy to build and easy to sell
Answer: B. A thin wrapper around a single API is exactly what the model provider can absorb (get 'Sherlocked') overnight. Easy to build often means easy for the platform to eat. Defensibility has to come from something the platform won't replicate.
Q5. What does 'graceful failure' mean in human-AI interaction design?
The AI should never make mistakes
Errors should be hidden from the user to preserve trust
The user can always notice, correct, or undo what the AI did
The AI should apologize whenever it produces output
Answer: C. Graceful failure means designing for the wrong answer up front: drafts the user approves rather than actions taken behind their back, an easy edit, a clear undo. That turns a mistake into a shrug instead of a disaster.
Practice
Exercise 1. Take an AI product idea and ask the Feature Graveyard test: if Anthropic or OpenAI shipped this as a built-in feature next quarter, would you still have a business? Name one source of defensibility (data, workflow integration, niche, community, expertise) that would survive that. (Hint: If your only answer is 'mine would have a nicer UI,' that's not defensibility. A nicer wrapper is the easiest thing for the platform to out-distribute.)
Exercise 2. Interview one real person about a problem you think is worth solving. Ask only about how they currently handle it, and don't pitch your idea. Write down any workaround they mention. (Hint: If they describe no workaround at all, that may mean the pain isn't as sharp as you assumed. That's a useful finding too.)
Exercise 3. Take an app idea and write its MVP scope as a list of explicit non-goals (Chapter 2), what you're deliberately not building for version one, and why. (Hint: For each cut feature, write one sentence on what evidence would change your mind and bring it back into scope.)
Sources
Graham, Why Smart People Have Bad Ideas: good at solving problems, bad at choosing them (April 2005) (https://paulgraham.com/bronze.html)
Ries, The Lean Startup: build-measure-learn and the minimum viable product (2011) (https://theleanstartup.com/principles)
Design Council (UK), the Double Diamond framework for innovation (2004) (https://www.designcouncil.org.uk/our-resources/framework-for-innovation/)
Fitzpatrick, The Mom Test: how to talk to customers when everyone is lying to you (2013) (https://www.momtestbook.com/)
Andreessen, The Only Thing That Matters: product/market fit (June 2007) (https://pmarchive.com/guide_to_startups_part4.html)
Kamepalli, Rajan and Zingales, Kill Zone (NBER Working Paper 27146, 2020) (https://www.nber.org/papers/w27146)
Apple's Sherlock 3 and Karelia's Watson: the origin of 'Sherlocked' (https://en.wikipedia.org/wiki/Sherlock_(software))
Graham, How to Get Startup Ideas: why the best ideas come from problems you already have (November 2012) (https://paulgraham.com/startupideas.html)
13. Deployment and Operations
Hosting, cloud, monitoring, scaling, and cost.
An app that only runs on your laptop isn't a product. This chapter covers shipping to the world: hosting and cloud infrastructure, monitoring, scaling, and keeping costs (especially AI API costs) under control.
By the end of this chapter, you will be able to
Map an app's frontend, backend, and database onto appropriate hosting choices
Separate environments and keep secrets out of source control
Set up basic monitoring and alerting for a deployed application's critical paths
Estimate and control the AI API cost of a feature before it scales
13.1 Hosting and cloud infrastructure
An app that only runs on your laptop is a project, not a product. The moment you close the lid, it's gone. Hosting is the step that makes it real: running your frontend, backend, and database on infrastructure that's reachable on the public internet, around the clock, whether or not your machine is on. This is the boundary between 'I built something' and 'people can use something I built.'
Modern cloud platforms (Vercel, Netlify, Render, the big three clouds, and the all-in-one platforms from Chapter 6) handle most of the gnarly underlying infrastructure for you (servers, networking, certificates), which is why deployment that used to take a sysadmin a day can now take a beginner minutes and leaves you with decisions instead of plumbing. The cloud, as the sticker says, is just someone else's computer. The rest of this chapter is about being a good tenant on it.
The main decision is how your pieces map onto that infrastructure. A static frontend (just HTML/CSS/JS) can be served cheaply from a CDN. Your backend can run as a long-lived server process or as serverless functions, small on-demand functions that spin up per request and that you pay for per execution, which is cheap at low traffic and elastic under spikes. The database is typically a managed service so you're not babysitting it. Each piece has its own deploy step, and knowing which piece is which keeps deployment from feeling like a black box.
A grounding note for this AI-heavy course: the AI parts of your app aren't special at deploy time. A model API call is just an outbound HTTPS request from your backend, so 'how do I deploy my AI app' is really just 'how do I deploy my app,' with the extra care that the API key lives safely server-side (next section) and the bill is watched (last section).
Commit โ CI tests โ Build โ Deploy โ Monitor
From a commit to a live site. Each stage is a chance to catch a problem before your users do.
What 'running in production' actually requires beyond the code itself.
13.2 Environments, secrets, and CI/CD
The single most useful operational habit is separating environments. Development is where you build, staging is a production-like copy where you test changes safely, and production is what real users touch. The whole point is that a change proves itself somewhere harmless before it can break things for actual people. Shipping straight from your editor to production is how a typo becomes an outage.
Secrets demand their own discipline, and it's non-negotiable: API keys, database passwords, and tokens go in environment variables or a dedicated secrets manager, never committed to source control [1], for an unforgiving reason. A secret pushed to git history is effectively leaked forever, because it lives in the repo's history even after you delete it from the current files. Bots actively scan public repos for exactly this, and once a key is exposed, the only real fix is to rotate it [2].
CI/CD (continuous integration and continuous deployment) automates the path from 'code pushed' to 'change live.' On every push, it can run your test suite (the tests from Chapter 10) and only deploy if they pass. This does two things at once: it removes the manual, error-prone deploy ritual, and it catches failures before users do instead of after, turning your tests from a thing you remember to run into a gate nothing broken can slip past.
These three together form a safety net that matters more in AI-assisted development, not less, precisely because you ship faster and read less of what you ship. When you're accepting large generated diffs at speed, the automated test gate, the staging buffer, and the discipline of never leaking a key are what stop that speed from translating directly into production incidents. And when something does slip through, a rollback (reverting to the last known-good deploy) is your fast undo.
13.3 SSH keys and secure access
Alongside API keys, there's a second kind of credential you'll set up early and then mostly forget about: the SSH key pair that authenticates you to GitHub and to remote servers without typing a password every time. It's a public/private key pair (generated once with ssh-keygen), where the private key stays on your machine, never shared, and the public key is the thing you paste into GitHub's settings or a server's authorized-keys list. The server can verify you own the private key without you ever sending it anywhere, which is the whole appeal over a password, and this section treats that key as the credential it is. Optional Chapter P is the hands-on version, walking through generating a pair, loading it into ssh-agent, and getting it onto your GitHub account.
The rule that matters is symmetric with API keys: the private key never leaves your machine, full stop. It is not something you paste into a chat to debug a connection issue, commit into a repo 'just for this project,' or copy onto a shared or public computer. Anyone who obtains your private key can authenticate as you, to every server and repo that trusts it, and unlike a password, there's no login prompt to tip you off that it happened.
If you ever suspect a private key was exposed (a shared machine, a misconfigured backup, a repo it got committed to), the fix mirrors a leaked API key: don't try to judge whether it was actually misused, just revoke it. Remove the corresponding public key from GitHub's settings and any server's authorized-keys file [3], generate a fresh pair, and redistribute the new public key. A key you're not fully sure about is a key you should already consider replaced.
13.4 Monitoring
Once real users are on your app, a hard truth sets in. It will break in ways you didn't anticipate, on inputs and conditions you never tested. The question is whether you'll find out before your users do. Monitoring is the answer. It's the difference between learning about an outage from a dashboard at 2:01 and learning about it from an angry support ticket at 9am after a night of broken checkouts.
The minimum viable setup is small and high-leverage. First, send errors somewhere you'll actually see them (a logging or error-tracking service) rather than to a server log file nobody opens. Second, put an alert on your critical paths: the handful of flows (login, checkout, the core action your product exists for) whose failure means the product is effectively down, so you don't need to monitor everything to start, only the things that matter.
The broader version of this is observability: being able to understand what your system is doing internally from its logs, metrics, and traces, so that when something is wrong, you can work out why it broke and not only that it broke. The distinction is practical: monitoring tells you the checkout is failing. Observability helps you see it's failing because the payment provider is timing out. You grow into this as the app grows. You don't need it all on day one.
For an AI-powered app, monitoring earns a special line item: watch your model usage and spend, not just your errors. An AI feature can be technically 'up' (no errors, all green) while quietly burning money or drifting in quality, and neither shows up as a crash. Treating cost and output quality as things to monitor, not just uptime, is what keeps a working AI feature from becoming an expensive surprise, which is exactly where the next section goes.
13.5 Scaling
Scaling is handling more users or more load without the system falling over. It's worth saying plainly up front: most projects, and nearly all student projects, never need to think hard about this, and prematurely engineering for millions of users you don't have is its own classic mistake. The goal here is literacy (knowing the levers exist and roughly what they are), not building for scale you'll probably never see.
The basic levers are few. Horizontal scaling means handling more load by running more instances of your service behind a load balancer, rather than buying one ever-bigger machine (vertical scaling), and it's how systems handle large, variable traffic gracefully. The database is the part that most often becomes the bottleneck first, since many app servers can pile concurrent queries onto one database, which is why managed databases and connection limits matter as you grow.
Caching is the highest-leverage scaling tool and worth understanding even on a small app: store the result of expensive or frequently repeated work so you can serve it again without redoing it. If a thousand users request the same page, computing it once and reusing the result is the difference between a calm server and a melting one, and caching also directly cuts cost, which connects it to the next section.
The reason to know any of this before you need it is that scaling problems arrive as a spike, not a memo. A post goes viral or a class assignment comes due at midnight, and a system with no headroom turns a moment of success into an outage at exactly the wrong time, so while you don't have to build for scale you don't have, you do want to recognize the pattern so a good problem ('lots of users!') doesn't become a bad night.
13.6 Cost management
Everything in the cloud costs money (compute, storage, bandwidth), but for AI apps there's one line item that surprises people more than any other, which is the model API bill, and the trap is structural. During development, you make a few test calls, and the cost rounds to zero, so it never registers as a concern. Then real traffic arrives, and that per-call cost (Chapter 7's token pricing) multiplies by every user and every request, linearly, with no warning.
Two specific patterns turn a reasonable bill into a shocking one. An inefficient prompt (stuffing a huge document into context for a one-line answer, or re-sending the whole conversation on every keystroke) pays for tokens you didn't need on every single call. And an unbounded agent loop (the runaway loop from Chapter 8) can spend an unbounded amount on a single task that isn't converging. Both stay invisible until the invoice arrives, which is why they bite.
The defenses are practical and worth building in from day one, not bolting on after the scare. Track spend from the start so the number is never a mystery. Set usage alerts or hard budget caps so a runaway process trips a wire instead of running all weekend, and choose the smallest model that clears the bar for each task. Using a frontier model to classify a sentence is like chartering a jet to cross the street, and Chapter 1's model-tier point pays off directly here.
Step back, and cost is really just one more dimension of engineering judgment this book keeps returning to, alongside correctness, security, and reliability, now joined by whether a feature is economically sane at scale. A feature that works beautifully and costs more per use than users will ever pay is a demo with a hidden time bomb. Treating cost as a first-class design constraint is part of what separates a shippable product from an impressive prototype.
Summary
Shipping is where a project meets real users and real bills. We covered hosting, environments and secrets, monitoring, scaling, and cost management. AI API usage is a cost worth estimating explicitly. The recurring discipline: keep secrets server-side, alert on the paths that matter, and know what would break first under load before it does.
Key terms
Hosting. Running your app on infrastructure reachable on the public internet, not just your local machine.
Cloud. Third-party infrastructure (Vercel, Netlify, Render, the big three clouds) that handles servers, networking, and certificates for you, turning deployment that used to take a sysadmin a day into something a beginner can do in minutes.
Serverless. Running backend code as on-demand functions without managing servers yourself. You pay per execution.
Environments. Separate copies of your app (development, staging, production) so changes are tested before reaching real users.
CI/CD. Continuous Integration / Continuous Deployment, automation that runs tests and ships changes whenever code is pushed.
Secrets. Sensitive values (API keys, passwords) kept in environment variables or a secrets manager, never committed to source control.
SSH key. A public/private key pair used to authenticate to GitHub or a remote server without a password. The private key never leaves your machine; a leaked one should be revoked and replaced, like any other credential.
Monitoring. Tracking errors and key metrics so you learn something broke before your users have to tell you.
Observability. The broader ability to understand a system's internal state from its logs, metrics, and traces.
Horizontal scaling. Handling more load by adding more instances of a service rather than making one machine bigger.
Caching. Storing the result of expensive or repeated work so it can be reused, reducing load and cost.
Cost management. Treating AI API spend as a first-class design constraint: tracking spend from day one, setting usage alerts or hard budget caps, and choosing the smallest model that clears the bar for each task.
Token cost. The per-token price of AI API usage, the line item that most often surprises teams as traffic scales.
Rollback. Reverting to a previous known-good deployment when a new release breaks.
Check your understanding
Q1. Where should API keys and database passwords live?
Committed directly into the source code for convenience
It doesn't matter as long as the repo is private
In the frontend so they're easy to access
In environment variables or a secrets manager, never in the repo
Answer: D. A secret committed to git history is effectively leaked forever, even if removed later. Secrets belong outside source control entirely.
Q2. What should you do if you suspect your SSH private key was exposed?
Revoke the public key everywhere it's trusted and generate a new pair
Nothing, since SSH keys can't really be misused like a password
Wait to see if anything suspicious happens on your accounts first
Change your GitHub account password and carry on as normal
Answer: A. A private key authenticates as you with no login-prompt tip-off if misused. Treat a suspected exposure like a leaked API key: revoke and replace it rather than waiting to confirm harm.
Q3. What does CI/CD primarily automate?
Writing the application source code on your behalf
Running tests and deploying changes whenever code is pushed
Designing and laying out the user interface
Generating the project documentation automatically
Answer: B. CI/CD removes manual, error-prone deploy steps by running checks and shipping changes automatically on push.
Q4. Why does the AI API bill surprise people most when scaling a product?
AI APIs switch to a flat monthly rate once you reach production
AI API costs stay entirely fixed no matter how much usage you drive
Negligible dev usage scales linearly with real traffic, and bad prompts compound
It doesn't surprise anyone, since AI APIs are the cheapest line item
Answer: C. A few test calls during development hide how costs scale with real traffic, and an unbounded agent loop can multiply that quickly.
Q5. What's caching, and why does the chapter call it the highest-leverage scaling tool?
Buying one ever-bigger machine to handle more load
Running more instances of a service behind a load balancer
Limiting the number of concurrent database connections
Storing the result of expensive work so it's served again for free
Answer: D. Caching reuses a computed result instead of redoing the work on every request, which reduces both server load and cost. Running more instances is horizontal scaling; buying a bigger machine is vertical scaling, a different lever.
Practice
Exercise 1. Deploy a small app to a free hosting tier end to end (frontend + backend, if applicable). Set up one basic alert or log you'd actually see if it broke. (Hint: Trigger an error on purpose after deploying and confirm you actually get notified. An alert you never tested isn't a real alert.)
Exercise 2. Estimate the monthly AI API cost of a feature you've built (or sketched) at 100 users making 10 requests/day each. State your assumptions about token usage per request. (Hint: Check the provider's published per-token pricing and multiply through deliberately. The goal is forming the habit of estimating before you ship, not precision.)
Sources
The Twelve-Factor App: store config in environment variables, never in code (https://12factor.net/config)
GitHub docs: removing sensitive data, and why rotating the secret comes first because history rewrites do not un-leak it (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository)
GitHub docs: reviewing and deleting SSH keys from your account (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/reviewing-your-ssh-keys)
14. Security, Ethics, and the Future of AI
Privacy, copyright, prompt injection, and responsible AI.
The closing chapter zooms out to responsibility. Privacy, copyright, prompt injection, and data security aren't optional add-ons. The future of the field belongs to people who build it thoughtfully.
By the end of this chapter, you will be able to
Apply a privacy checklist before sending user data to a third-party AI API
Red-team your own application for authorization and prompt-injection vulnerabilities
Explain the unsettled copyright questions around AI training data and AI-generated output
Apply the 'who could this hurt' question to a system before shipping it
14.1 Privacy
Privacy is the practice of handling user data with deliberate limits on who can see it, how long it's kept, and what it's used for, applying to data that feels mundane and not just to obvious secrets. An email address, a location, and a list of someone's searches are all things people reasonably expect you not to spray around. The mental shift is treating user data as borrowed. You hold it for a purpose and with limits, not as a free resource to do whatever you like with.
AI building adds a specific question that's easy to skip in the moment: when you send data to a model API, where does it go, and what happens to it there? Providers differ enormously in what they retain, so the answer is never automatic. Some don't train on API inputs by default [10] but still store them for a window before deletion [12], some train on them unless you opt out, and some keep the data far longer. Sending real user data to a third party without knowing that policy is a privacy decision you've made by default, whether or not you meant to make one.
Extra care is owed to PII, personally identifiable information like names, emails, addresses, anything that pins data to a specific human. Before piping PII through an AI feature, it's worth asking whether you even need to: often you can strip or pseudonymize the identifying parts and send only what the task requires, which limits the damage if anything ever goes wrong, since the data you never sent is the data that can never leak.
None of this requires becoming a lawyer. It requires forming a habit. Before a new data flow, pause on three questions. What data is this? Where is it going? Does the recipient's policy match what my users would expect? That thirty-second check is the difference between a considered privacy posture and finding out the hard way that 'we sent everything to a third party and never looked at their terms' was your policy all along, an ugly policy but a real one.
The main places a small app gets attacked. Most real incidents trace back to one of these.
As tools take over more of the 'how,' accountability stays firmly on your end of the bar.
14.2 Data security
Security has run as a quiet thread through this whole book, and here it gets named directly. The fundamentals don't change because AI is involved: authentication and authorization enforced correctly (Chapter 6), secrets kept out of source control (Chapter 13), and, the one principle under all of them, input treated as untrusted until you've validated it, because a user, or anyone pretending to be one, can send your system anything, and assuming otherwise is where most vulnerabilities begin.
What AI changes is the tempo, and that cuts against you. When you can generate a new endpoint, a new form, a new data flow in minutes, you also generate new attack surface in minutes, and a model focused on making a feature work will reliably produce something functional that quietly skips a validation check or an authorization rule (the exact pattern from Chapter 6), so velocity multiplies both features and holes.
So security can't be a single pass at the end. It has to be a repeated, deliberate step that keeps pace with how fast you're shipping. The concrete habit is to interrogate each new surface as it appears: who can call this, what happens if they send something malformed or malicious, does this respect the same access rules as everything else. Asking those questions per-feature, while the feature is fresh, is far cheaper than discovering the answer in an incident.
The single highest-value practice, red teaming, is to attack your own system before someone else does. Deliberately try the things an attacker would: send another user's ID and see if you get their data, submit a giant or malformed payload, attempt the action you're not supposed to be allowed. This is the same 'break your own access rules' move from Chapter 6, generalized into a mindset. Finding the hole yourself is a fix. Having a stranger find it is a breach.
14.3 Copyright and AI output
Copyright is where building with AI runs into law that hasn't caught up, and honesty about the uncertainty is more useful than false confidence. Two distinct questions come up here, and they are worth keeping separate. First, the input side: what's the legal status of the training data a model learned from, some of which was copyrighted work used without explicit permission? That's the subject of active, high-stakes litigation whose outcome isn't settled. One court rejected a fair-use defense for training on copyrighted legal headnotes [6]. The clearest picture so far comes from the case against Anthropic, because a single ruling went both ways. Anthropic had bought millions of used print books, cut the bindings off, scanned the pages, and thrown the paper away. In June 2025 the court held that this was fair use, on the reasoning that buying a book and converting it to a searchable digital copy you do not redistribute is a change of format rather than a new copy [11]. The same ruling found that downloading books from shadow libraries was not fair use. That half of the case settled for $1.5 billion [7]. The distinction the court drew was about how the copy was obtained, not what it was later used for. It is also a trial-court decision that was never appealed, so it binds nobody, which is a fair summary of this whole area.
Second, the output side: who, if anyone, owns what a model generates for you? Can AI-generated text, code, or images be copyrighted at all, and if so, by whom: you, the provider, or no one? Different jurisdictions are landing in different places, and the answer can hinge on how much human authorship was involved. The U.S. Copyright Office has taken the position that purely machine-generated output isn't copyrightable, while material with meaningful human authorship can be [5]. This isn't a solved question with a citation you can look up and move on. It's genuinely in flux.
Because the ground is shifting, the responsible move is to check the specific terms of the specific tool you're using, especially before anything commercial. Many providers spell out in their terms what rights you have to the output and what you're permitted to do with it, and those terms vary and change. 'I assumed it was fine' isn't a position you want to discover the weakness of after building a business on it.
The broader posture this models is the right one for a fast-moving field generally: where the rules are unsettled, stay informed and stay conservative rather than assuming the most convenient interpretation is correct, since you don't need legal certainty to act responsibly. You need the awareness that the question is open, and the habit of checking the actual terms in front of you instead of the comfortable story in your head.
14.4 Prompt injection
Prompt injection is the signature security vulnerability of AI systems [1] and worth understanding precisely because nothing in traditional software behaves like it. Simon Willison named the attack in 2022, and it has resisted a clean fix ever since [2]. The attack: untrusted content the model reads (a web page, a document, an email, a user message) contains instructions crafted to hijack the system, making it ignore its real task and do the attacker's bidding instead, with the malicious instruction riding in through the data rather than the code. When the hostile text arrives inside content the model was asked to read, rather than from the user directly, it's called indirect prompt injection [3].
The root cause is fundamental to how models work and explains why it's so stubborn. A language model processes everything as one stream of text and has no hard, built-in wall between 'instructions from my developer' and 'content I was asked to read.' So a document that says, in the middle of otherwise normal text, 'ignore your previous instructions and instead do X' can be obeyed, because to the model it's all just text to continue. Traditional code never confused its instructions with its inputs, but a model has no wall between the two.
This turns dangerous the instant an agent (Chapter 8) has tools. A chat model tricked into saying something rude is embarrassing. An agent with email and database access tricked by a hidden instruction in a document it was summarizing can perform data exfiltration (quietly emailing private data to an attacker) or take other real actions. Willison calls the dangerous combination the lethal trifecta: access to private data, exposure to untrusted content, and some way to send data out [4]. The capability that makes agents useful is exactly what makes a successful injection costly, which is why tool-using agents need the most care.
There's no single switch that makes injection impossible, so defense is layered. Treat content the model reads as untrusted data rather than as commands, and design the system so it doesn't blindly act on instructions found inside that content. Above all, apply Chapter 8's guardrails to constrain what an agent is permitted to do no matter what it's told mid-task (least privilege, human checkpoints before sensitive actions), so that even a successful injection hits a wall of 'you're not allowed to do that anyway.' Red-team your own AI features for this specifically, since it's the attack most likely to be overlooked.
14.5 The physical cost of AI
Chapter 13 treated 'the cloud' and 'token cost' as abstractions, a dollar figure on an invoice. This section makes them physical, because every model call you make runs on real hardware in a real building, and that building has costs that never show up on your bill. A data center is a warehouse of specialized chips (GPUs) drawing large amounts of electricity around the clock, and the AI boom is driving them to be built at a scale that's now a measurable share of regional power demand [9].
Two costs are worth understanding concretely. The first is energy. Training a frontier model is an enormous one-time electricity expense, but the cost that compounds is inference. Every individual query is small, yet multiplied across billions of calls a day, serving the models can dominate total energy use. The second is water: many data centers use evaporative cooling, consuming significant fresh water [8], which becomes a genuine local conflict when a facility is sited in a drought-prone or water-stressed region. There's also embodied cost upstream, the carbon and mining behind manufacturing the chips, and the e-waste when they're retired.
Where the harm actually lands is the part that connects to the rest of this chapter: it's local and uneven. A data center can strain a regional electrical grid, raise local energy prices, delay the retirement of fossil-fuel plants, or draw down an aquifer a community depends on, costs borne by people who never used the AI and never consented to the trade. That's the same 'who bears the cost when the system works as designed' question from 14.2 and the responsible-AI section, just pointed at infrastructure instead of data.
The honest framing is a trade-off. This isn't an argument to stop using AI, and the numbers get wildly exaggerated in both directions, so be skeptical of any single dramatic statistic, though what the trade-off does change is how you build. The model-selection instinct from Chapter 13 (use the smallest model that clears the bar) and the cost-efficiency habits there aren't only about your wallet. An inefficient prompt or a needless frontier-model call spends real energy and water too. Efficiency, the provider's energy mix, and not reaching for a giant model when a small one suffices are where an individual builder's choices actually touch this, small per call, real at scale.
14.6 Open weights, regulation, and who gets the gains
The questions in this chapter apply one level up, to the models themselves, and the sharpest version is about open weights. An open-weights model is one whose trained parameters anyone can download and run (Optional Chapter H shows how), and whether to release them is a live argument. The case for is strong. Open models cost nothing beyond the hardware to run them, they let researchers inspect what closed vendors keep private, and they put capable AI in the hands of people and countries who could never pay frontier API prices [13]. That access matters, because the IMF estimates that about 40 percent of jobs worldwide are exposed to AI and warns that without deliberate policy the gains will concentrate among those who already hold capital and access, widening the gap between rich and poor within countries and between them [14]. The case against is that a released model can't be recalled or patched. Whatever it can do, including helping with cyberattacks or biological weapons, it can do for anyone, permanently, and the safety training a vendor added can be stripped out by anyone willing to fine tune it [13].
Where to draw the line is unsettled, and the people drawing it disagree in public. The US government's 2024 review concluded that the evidence didn't yet justify restricting open weights and called for monitoring instead [15]. The EU's AI Act exempts open source models from some of its documentation duties, but not models it classifies as carrying systemic risk [16]. Anthropic, which does not release its own weights, has said it doesn't support banning open models and argues instead for mandatory safety testing of any sufficiently capable model, open or closed, along with controls on the chips used to train them [17]. The reason to hold this in mind as a builder is that the same trade shows up in your own choices. A local open model keeps your users' data on your machine and costs nothing per call, and it also arrives with no vendor watching for misuse, so the responsibility a hosted provider shares with you becomes entirely yours.
14.7 Responsible AI and the road ahead
The course ends by widening the lens from 'does it work' to 'should it exist, and who might it hurt.' Responsible AI development means anticipating harm before you ship, which goes past verifying that the product works for the user you had in mind. The hard part is that a system can pass every test you wrote and still cause real damage, because the harm lands on people and cases you never thought to test.
The examples are concrete and current. A hiring tool can run flawlessly and systematically disadvantage a group, because bias in its training data became bias in its decisions. A content recommender can maximize engagement exactly as designed and still pull people toward extremes, and a chatbot can be helpful to most users while harming a vulnerable one. In each case the failure isn't a bug in the usual sense. The code did what it was told, and what it was told had consequences nobody examined. This is why bias and broader alignment (getting systems to act in line with human intent and values) are engineering concerns, not just philosophical ones.
The discipline that addresses this is a single repeatable question asked before shipping: who could this hurt, and how would we know if it did? That question forces you past your intended happy path to the people at the edges: the misuse case, the group underrepresented in your data, the user in a situation you didn't imagine. It won't surface every harm, but the teams that ask it routinely catch the harms that teams who never ask ship straight into the world.
That habit is the real graduation requirement of this book, because it's the one thing that doesn't expire. The specific tools here will be outdated within a few years. That's the nature of this field, and Optional Chapter L shows how every past wave of AI looked permanent and wasn't. What carries forward is judgment: the instinct to specify clearly, verify relentlessly, and ask who bears the cost when you're wrong. Build things people actually need, stay honest about what you've confirmed versus assumed, and take responsibility for what you ship, which is vibecoding done right and yours to carry past this page.
Summary
The last core chapter turns the verification habit outward to consequences. We covered privacy and data handling, prompt injection where hidden instructions hijack an agent, and copyright. We covered the physical cost of AI and the 'who could this hurt' review. The durable takeaway: the specific tools will change, but treating untrusted input as untrusted and asking who bears the risk is the skill that outlasts them.
Key terms
Privacy. Handling user data with deliberate limits on who sees it, how long it's kept, and what it's used for.
PII (Personally Identifiable Information). Data that can identify a specific person (names, emails, addresses), requiring extra care before sending to any third party.
Data retention. The length of time a provider stores your inputs and whether it uses them for training, the policy you must check before sending real user data.
Data security. Treating all input as untrusted until validated, and enforcing authentication and authorization correctly, as a repeated per-feature habit rather than a single pass at the end, since AI-assisted speed generates new attack surface as fast as it generates new features.
Copyright. The unsettled legal questions around training-data rights and ownership of AI-generated output.
Open-weights model. A model whose trained parameters are published for anyone to download, run, and modify. Cheap and inspectable, and impossible to recall once released.
Prompt injection. An attack where untrusted input contains instructions that hijack an AI system into ignoring its real task.
Jailbreak. A prompt crafted to bypass a model's safety guardrails and make it produce restricted output.
Data exfiltration. Unauthorized extraction of data, a key risk when prompt injection targets an agent with tool or data access.
Alignment. The effort to make AI systems behave in line with human intent and values.
Bias. Systematic unfairness in a model's behavior, often reflecting skews in its training data, that can harm specific groups.
Red teaming. Deliberately attacking your own AI system (injection, jailbreaks, abuse) before shipping to find weaknesses first.
Responsible AI. Anticipating who a system could harm, including people outside your test cases, before you ship it, not after.
Inference vs. training cost. Training a model is a large one-time energy expense. Inference (serving each query) is small per call but, at billions of calls, can dominate AI's total energy use.
Water footprint. The fresh water consumed by data-center cooling, a real local-resource concern when facilities sit in water-stressed regions.
Embodied carbon. The emissions baked into manufacturing hardware (chips, servers) before it ever runs, plus the e-waste when it's retired.
PUE (Power Usage Effectiveness). A data-center efficiency metric: total facility energy divided by the energy actually delivered to computing, where closer to 1.0 is more efficient.
Check your understanding
Q1. Before sending real user data to a third-party model API, what should you check?
The provider's data retention and usage policy for that data
Nothing, all AI providers handle data identically
Only whether the API call is free
Whether the data is formatted as JSON
Answer: A. Providers differ in whether and how long they retain inputs or use them for training. Checking the policy is the actual privacy decision.
Q2. What makes prompt injection especially dangerous for agents with tool access?
Agents with tool access are immune to prompt injection
Content it reads can hide instructions that hijack its tool calls
It affects only chat assistants that have no tools attached
Prompt injection only works through image inputs, not text
Answer: B. An agent that can act (send data, call tools) turns a hijacked instruction into a real-world action, not just a bad text response.
Q3. What does 'responsible AI development' mean in practice, per this chapter?
Testing with your intended users and treating that as sufficient
Avoiding AI in any product that touches people
Anticipating who a system could harm before you ship it
Responsibility sits with the AI provider, not with the builder
Answer: C. A system that works for your test cases can still harm people you never tested with at scale. Anticipating that gap is the core habit.
Q4. At global scale, which part of an AI system's energy use tends to dominate over time?
Training the model, since it's by far the largest single event
Neither, since data centers use a negligible amount of energy
Writing and running the code that calls the model
Inference, since each query is small but billions of them aren't
Answer: D. Training is a big one-time cost, but inference recurs on every single query. At billions of calls a day, serving the model can outweigh the cost of training it.
Q5. Given that AI copyright law is still unsettled, what does the chapter recommend before using AI output commercially?
Check the terms of the specific tool, since rights vary and change
Assume AI-generated output is automatically copyrighted by you
Assume the question is fully resolved and no longer worth checking
Avoid AI-generated output entirely in any commercial product
Answer: A. Both the training-data question and the output-ownership question are genuinely in flux and vary by jurisdiction and provider. The responsible move is checking the actual terms in front of you rather than assuming the most convenient interpretation.
Practice
Exercise 1. Take one feature you've built that calls an AI model. Identify one change that would cut its energy/water footprint (e.g. a smaller model, a tighter prompt, caching repeated calls) and explain why it helps. Note that the same change usually lowers your dollar cost too. (Hint: This is the same model-selection and efficiency thinking as Chapter 13's cost management. The physical cost and the financial cost move together.)
Exercise 2. Find the data retention/training-use policy for one AI API you've used while working through this book. Summarize it in two sentences and note whether it would change how you'd use it with real user data. (Hint: Provider policies are usually in a 'data usage' or 'privacy' page in their developer docs, not the general consumer privacy policy.)
Exercise 3. Design a simple prompt-injection attack against a hypothetical agent that reads emails and can send replies on your behalf. Then propose one concrete defense. (Hint: Think about what a malicious email could say to make the agent take an action you didn't ask for. Then think about treating email content as data, not instructions.)
Sources
OWASP GenAI Security Project, Top 10 for LLM Applications: LLM01 Prompt Injection (https://genai.owasp.org/llm-top-10/)
Willison, Prompt injection attacks against GPT-3: the post that named the attack (September 2022) (https://simonwillison.net/2022/Sep/12/prompt-injection/)
Greshake et al., Not what you've signed up for: compromising LLM-integrated applications with indirect prompt injection (AISec workshop at ACM CCS, 2023) (https://doi.org/10.1145/3605764.3623985)
Willison, The lethal trifecta for AI agents: private data, untrusted content, and exfiltration (June 2025) (https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/)
U.S. Copyright Office, Copyright and Artificial Intelligence, Part 2: Copyrightability (January 2025) (https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-2-Copyrightability-Report.pdf)
Thomson Reuters v. Ross Intelligence, summary-judgment opinion rejecting fair use, Bibas, J. (D. Del., February 2025; argued on appeal in the Third Circuit June 2026, No. 25-2153, decision pending) (https://www.ded.uscourts.gov/sites/ded/files/opinions/20-613_5.pdf)
Authors Guild, Court grants final approval of the $1.5B Bartz v. Anthropic copyright settlement (July 2026) (https://authorsguild.org/news/court-grants-final-approval-anthropic-copyright-settlement/)
Li et al., Making AI Less 'Thirsty': the water footprint of AI (Communications of the ACM, 2025) (https://doi.org/10.1145/3724499)
IEA, Key Questions on Energy and AI: data-center electricity demand and projections (April 2026) (https://www.iea.org/reports/key-questions-on-energy-and-ai)
Anthropic privacy docs: commercial inputs and outputs not used for training by default (https://privacy.claude.com/en/articles/7996868-is-my-data-used-for-model-training)
Bartz v. Anthropic, order on fair use, Alsup, J. (N.D. Cal., June 23, 2025): purchased-and-scanned books fair use, shadow-library downloads not (https://fingfx.thomsonreuters.com/gfx/legaldocs/jnvwbgqlzpw/ANTHROPIC%20fair%20use.pdf)
Anthropic privacy docs: how long commercial inputs and outputs are stored before deletion (https://privacy.claude.com/en/articles/7996866-how-long-do-you-store-my-organization-s-data)
Kapoor et al., On the Societal Impact of Open Foundation Models: the benefits and marginal risks of releasing model weights (ICML 2024) (https://proceedings.mlr.press/v235/kapoor24a.html)
Cazzaniga et al., Gen-AI: Artificial Intelligence and the Future of Work: 40 percent of global employment exposed to AI, and the inequality risks (IMF Staff Discussion Note 2024/001, January 2024) (https://www.imf.org/en/Publications/Staff-Discussion-Notes/Issues/2024/01/14/Gen-AI-Artificial-Intelligence-and-the-Future-of-Work-542379)
NTIA, Dual-Use Foundation Models with Widely Available Model Weights: the US review that recommended monitoring rather than restricting open weights (July 2024) (https://www.ntia.gov/programs-and-initiatives/artificial-intelligence/open-model-weights-report)
EU Artificial Intelligence Act, Article 53: obligations for providers of general-purpose AI models, and the open-source exemption that does not cover systemic-risk models (https://artificialintelligenceact.eu/article/53/)
Anthropic, Our position on open-weights models: no ban, mandatory safety testing for capable models open or closed, and chip controls (July 2026) (https://www.anthropic.com/news/position-open-weights-models)
Optional A. Pricing and Funding an AI Startup
Pricing as positioning, unit economics, runway, and whether to raise.
Building the product is one problem. Paying for it is another. This chapter covers what to charge, what each customer actually costs you once inference is a variable cost, and what taking outside money commits you to.
By the end of this chapter, you will be able to
Set a price and explain what it signals about who the product is for
Estimate gross margin for a product whose inference cost scales with usage
Explain what raising outside money commits a company to, and when growing on revenue is the better instrument
A.1 Pricing as positioning
Charging money is the sharpest demand signal available to an early product. A free sign-up costs the person nothing and proves close to nothing, while a card entered proves that someone values the thing more than the dollars it costs them. Even a token amount works, because the signal is the act of paying and not the size of the payment. Founders delay pricing because a price invites a no, which is precisely the reason to ask early. Only a real price separates someone who would pay for this from someone who merely likes it.
A price is also a claim about who the product is for [1]. A few dollars a month says personal tool, bought on a whim and canceled on one. A four-figure annual contract says business system, with a budget owner and an approval step behind the purchase. Neither claim is more honest than the other, so the mistake is making one by accident. Underpricing is the more common accident and the more expensive one, because it starves the business of the margin it needs while telling buyers the product is cheap. Find the number by testing it against what a real buyer will pay, not against what feels comfortable to say out loud.
For AI products, the shape of the price is as much of an open question as its size. A survey of AI companies in mid-2026 found a subscription component still the most common arrangement, with consumption-based pricing rising from 35 to 42 percent of companies over six months, outcome-based pricing rising from 18 to 23 percent, and the average company running 1.7 pricing models at once [2]. Charging by usage or by result is harder to sell than a flat monthly fee, so the drift toward it is telling you something about what these products cost to run. Whichever shape you choose, write down what one customer pays you each month and keep that number next to what the same customer costs you to serve.
A.2 Unit economics
Unit economics is the arithmetic of one customer. Four numbers carry most of it. Acquisition cost is what you spend on sales and marketing to win a customer, divided by the customers you won. Churn is the rate at which customers leave, and its inverse tells you how long an average one stays. Lifetime value is the revenue a customer produces across that stay. Gross margin is the share of each revenue dollar left after the cost of serving that customer. Together they tell you whether a customer returns more than they cost and how long the return takes. A business whose acquisition cost exceeds its lifetime value gets worse with volume instead of better, because every new customer deepens the hole.
Classic software has near-zero marginal cost. Once the code is written, the thousandth user costs a rounding error more than the nine hundred ninety-ninth, which is why software businesses conventionally ran at eighty to ninety percent gross margin and why most of the SaaS playbook assumes they still do. An AI product breaks that assumption at the root, because every model call consumes compute that somebody bills you for, so cost to serve climbs with usage instead of flattening out. Surveyed AI companies averaged about 45 percent gross margin in 2025 and projected about 53 percent for 2026 [2], while one investor sample put the fastest-growing AI companies near 25 percent and the tier below them near 60 [3]. Those numbers describe the shape of the cost curve, not a failure of the operators sitting on it.
That lands directly on your price. A flat monthly fee holds revenue per customer constant while cost per customer is free to move, so a small fraction of heavy users can consume more compute than the whole cohort pays for. Utilization widens the gap further, because a measurement study across 42 benchmark configurations found effective cost per output token varying by a factor of 2.5 to 36 on identical hardware, depending on whether request volume kept the accelerator busy [4]. This is why coding assistants that launched on flat monthly plans now meter their expensive requests once an included allowance runs out [5]. Holding that cost down inside your own app is 13.6, watching your own token bill is Optional Chapter Q, and the current prices are in Optional Chapter M.
A.3 Raising money, or not
Raising money means selling a piece of the company. Equity is ownership, and dilution is what happens to your percentage when new shares are created for someone else. The common early instrument is a convertible one that sets no price when you sign it, converting into shares at the next priced round on terms fixed by a valuation cap [6]. These stack, and each one you sign sells a slice, so founders who sign several without doing the arithmetic first meet the combined effect at conversion, when it is no longer negotiable. The post-money version of the instrument exists to make that total calculable at the moment of signing, which is why its user guide leads with worked examples [6].
What you sell alongside the equity is optionality. A venture fund returns capital through a small number of enormous outcomes. Its arithmetic therefore requires the companies it backs to grow at a rate that could produce one [7], which makes accepting that money an agreement to pursue that rate. A company compounding quietly on its own revenue is a good outcome for its founders and a dead position inside a venture portfolio, and no amount of goodwill on either side resolves that mismatch once the money is in. Decide which kind of company you are building before you take capital that has already assumed an answer.
Runway is the number of months you can keep operating at your current burn before the money is gone. A raise buys time to reach a milestone you could not reach on revenue alone, so the useful order of questions is what the milestone is, what reaching it costs, and only then how much to raise. When revenue can already get you there, growing on revenue is the better instrument, because it costs you no ownership and no obligation to anyone. Optional Chapter J argues that the AI era pushed that threshold down, since a one-person company with low running costs needs fewer paying customers to clear its own bar.
Summary
What you charge and how you fund it decide whether a good product becomes a business. We covered pricing as a positioning decision, the unit economics of a product whose costs rise with usage, and the choice between growing on revenue and raising outside money. The recurring lesson: a price is a question worth asking early, and the answer changes what kind of company you can build.
Key terms
Unit economics. The arithmetic of a single customer, comparing what that customer pays you against what they cost you to win and to serve.
Customer acquisition cost (CAC). The sales and marketing spend of a period divided by the customers it won, meaning the price of one new customer.
Lifetime value (LTV). The total revenue one customer produces before they churn, set by what they pay each period and how long they stay.
Gross margin. The share of each revenue dollar left after the cost of serving the customer. AI products carry a lower one than classic software because inference, the compute billed on every model call, sits inside that cost and rises with usage.
Dilution. The drop in each existing owner's percentage of a company when new shares are issued to a new investor. Convertible instruments stack, so several small ones can add up to a large total at conversion.
Check your understanding
Q1. Why can a flat monthly price lose money on an AI product's heaviest users?
Because heavy users tend to churn faster than light users do
Because acquisition cost is always higher for heavy users than for light ones
Because storage and bandwidth dominate the cost of serving any modern application
Because inference is a variable cost that rises with usage
Answer: D. Classic software has near-zero marginal cost, so one flat price covers every user. Every model call bills real compute, so cost to serve climbs with usage while a flat fee holds revenue per customer still.
Q2. Besides collecting revenue, what does a price communicate?
How much the product cost to build in engineering hours
Which kind of buyer the product is meant for
Whether the underlying model provider is profitable at scale
How many competitors are currently in the same market category
Answer: B. A few dollars a month positions a product as a personal tool bought on a whim. A four-figure annual contract positions it as a business system with a budget owner behind the purchase.
Q3. What does taking venture money commit a founder to?
Reaching profitability within the first twelve months of operating
Hiring a full executive team before shipping the first product
Pursuing a growth rate that can produce an outsized outcome
Giving up day-to-day control of the product roadmap immediately
Answer: C. A fund returns capital through a small number of enormous outcomes, so its arithmetic requires portfolio companies to grow fast enough to produce one. Steady compounding on revenue is a good founder outcome and a dead position inside that portfolio.
Practice
Exercise 1. Take one product idea and write down the two numbers that decide whether it works: what a single customer pays you per month and what that same customer costs you to serve at their heaviest plausible usage. Then state whether a flat price survives that customer. (Hint: Estimate the cost side from the model calls a heavy user would trigger in a month, not from an average user. If the heaviest one costs more than they pay, decide now whether you cap usage, meter it past an allowance, or raise the plan price.)
Exercise 2. Name the one milestone a raise would buy you and what reaching it costs. Then work out how many paying customers at your chosen price would reach the same milestone without a raise, and pick the instrument. (Hint: If revenue can reach the milestone on its own, the ownership you would sell is the price of getting there sooner. Optional Chapter J's bootstrapping math is the comparison case.)
Sources
Y Combinator, Startup Pricing 101 (Kevin Hale, YC Library) (https://www.ycombinator.com/library/6h-startup-pricing-101)
ICONIQ Growth, 2026 State of AI: The Builder's Economy (July 2026) (https://www.iconiq.com/growth/reports/state-of-ai-2026)
Bessemer Venture Partners, The State of AI 2025 (August 2025) (https://www.bvp.com/atlas/the-state-of-ai-2025)
Patil, Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation (June 2026) (https://doi.org/10.48550/arXiv.2606.11690)
GitHub Docs, About premium requests: allowances and metered overage billing (https://docs.github.com/en/copilot/managing-copilot/monitoring-usage-and-entitlements/about-premium-requests)
Y Combinator, Safe financing documents and the Safe User Guide (https://www.ycombinator.com/documents)
Graham, Startup = Growth: why a venture fund's math requires a growth rate (September 2012) (https://paulgraham.com/growth.html)
Optional B. Information Pollution: What Mass AI Content Does to the Web
Dead Internet Theory, content farms, synthetic media, and the incentives behind them.
A deeper, societal look at what mass AI content does to the information ecosystem.
By the end of this chapter, you will be able to
Explain Dead Internet Theory and the economics behind content farms
Identify provenance signals that help distinguish trustworthy from synthetic content
Evaluate whether a product design rewards engagement or information quality
B.1 Dead Internet Theory and content farms
Dead Internet Theory began as a fringe conspiracy claim, that the internet 'died' years ago and is now mostly bots talking to bots [1]. Taken literally, it's overblown, but it has become a useful shorthand for something real: a growing and now-measurable share of online content, engagement, and even apparent accounts is automated, not human. Automated traffic passed half of all web traffic in 2024 and climbed above 53 percent in 2025 [4], and since early 2025 AI-generated articles have held at roughly half of everything newly published on the web, peaking at 50.9 percent in the last quarter of 2025 before settling back to 49.9 percent in the first quarter of 2026 [5]. You don't have to believe the conspiracy to notice the trend it points at.
Content farms are the clearest current face of it. These are operations that mass-produce low-quality articles (recipes buried under life stories, thin 'best of' listicles, SEO bait) purely to capture search traffic and ad revenue, and generative AI dropped their cost of production to nearly zero. What used to require a room of underpaid writers now takes one prompt and a loop, so the volume has exploded accordingly, and NewsGuard has tracked the resulting sites in the thousands [3].
This is the same coding-and-SEO slop from Chapter 9, viewed at the scale of the whole ecosystem instead of one bad output, where a single hallucinated article is a nuisance. A million of them, indexed and cross-citing each other, start to crowd out the human-written sources they were trained on. The harm lands on the shared information commons once slop becomes most of what fills it.
AI generates โ Published โ Scraped โ Trains next model โ (repeat)
The feedback loop behind information pollution: AI output becomes training data for the next model.
Where sources sit on the signal-to-noise axis as AI-generated content floods the web.
B.2 Synthetic media and incentives
Synthetic media (AI-generated text, images, audio, and video) is now cheap enough that the cost of creating content has effectively collapsed. That sounds like abundance, but it relocates the bottleneck: when anyone can produce infinite content, the scarce resource is no longer creation but attention, and everything optimizes for getting noticed rather than for being true or useful.
Crucially, the incentive structure isn't new. AI just supercharged it. Clickbait, engagement bait, and outrage farming existed long before generative models, because the underlying economics (ad impressions reward volume and provocation, not accuracy) were already there, so AI didn't invent the perverse incentive. It removed the last friction that was limiting how much slop the incentive could produce.
Understanding it as an incentive problem rather than a technology problem matters, because it tells you where solutions can and can't come from. You can't prompt your way out of a system that pays for attention regardless of quality. The leverage is in what gets rewarded: ranking signals, platform design, business models that pay for trust instead of clicks. Builders sit inside these incentives, so the question 'what's my product actually rewarding' is a design responsibility, not an abstraction.
B.3 Defending information quality
In a flooded ecosystem, the defensive skill shifts from finding information to judging it, and the key concept is provenance: where did this come from, who's behind it, and can it be verified? Chapter 11.4 covers C2PA, the standard for attaching that history to a file [6]. Content itself is now cheap to fake, so its source becomes the thing worth weighing. Favoring outlets and authors with a real track record, and treating anonymous or unverifiable claims with extra skepticism, is basic hygiene rather than cynicism.
Two concrete habits do most of the work. Cross-check anything surprising or consequential against an independent primary source. Chapter 9.2 explains why seeing a claim in several places is no longer proof of anything [2]. And weight behavior and evidence over confident assertion, the same calibration the whole book teaches, now applied to the open internet instead of a single model's output.
For builders, there's an active role beyond personal defense: you can build for quality or for slop, and the choice is rarely neutral. Tools that measure and surface trustworthiness over raw engagement, products that cite their sources, and systems that make provenance visible all push the ecosystem the other way. Every AI feature you ship either adds to the flood or helps people navigate it, and which one is a decision you're making whether or not you notice you're making it.
Summary
As AI-generated content floods the web, finding information gets easy, and judging it gets scarce. We covered the Dead Internet idea as a useful exaggeration and provenance as the thing worth weighing. We examined the builder's choice to surface trustworthiness over raw engagement. Every trust signal you ship becomes a new target to game. That's the design tension to hold.
Key terms
Dead Internet Theory. The claim, increasingly a useful shorthand, that a growing share of online content and activity is automated rather than human.
Content farm. An operation that mass-produces low-quality content purely to capture search or feed traffic.
Synthetic media. AI-generated text, images, audio, or video.
Incentives. The underlying economics (ad impressions rewarding volume and provocation, not accuracy) that produced clickbait and content farms long before AI. AI didn't invent the incentive. It removed the friction that limited how much slop it could produce.
Information quality. The degree to which content is trustworthy and verifiable, the thing a builder can choose to surface and reward instead of raw engagement.
Provenance. The traceable origin of a piece of content, where it came from and whether it can be verified.
Engagement bait. Content optimized to provoke clicks or reactions rather than to inform, the incentive that drives much slop.
Check your understanding
Q1. What incentive structure drives the production of content-farm slop?
Search and attention economics reward volume over accuracy
Content farms are required by law to publish a minimum volume
Slop is produced only by accident
There's no real incentive, it's random
Answer: A. When attention (clicks, ranking) is what gets monetized, producing more content cheaply beats producing fewer, more accurate pieces.
Q2. Why does the chapter argue that 'I saw it in several places online' is weaker evidence in an AI-saturated web?
Because search engines have stopped indexing multiple sources
Because those places may all trace back to one fabricated origin
Because online content has become reliably accurate on the whole
Because provenance metadata makes cross-checking unnecessary
Answer: B. AI-generated claims can be seeded across many sources that cite each other, so repetition doesn't guarantee independent confirmation.
Q3. What concrete role can a builder play in the information-pollution problem, according to this chapter?
None at all, since only regulation can meaningfully address it
Builders are only able to worsen it, and never to improve it
Choosing whether your product surfaces provenance or chases engagement
Refusing, personally, to use any generative AI tool for any purpose
Answer: C. Tools that cite sources and make provenance visible push the ecosystem one way; tools that optimize for raw engagement push it the other. Which one you ship is a design decision, not an abstraction.
Practice
Exercise 1. Find one online article you suspect is AI-generated content-farm slop. List three specific signals (not just a feeling) that led you to that conclusion. (Hint: Look for repeated filler phrases, a lack of any specific, checkable fact, or a byline with no other findable work.)
Exercise 2. Sketch a feature for a feed or search product that surfaces provenance (source, verifiability, track record) instead of raw engagement. Then write one sentence on how someone could try to game the signal you chose. (Hint: Every trust signal becomes a target once people know it's rewarded. If your signal has no obvious way to be gamed, look harder, because that usually means the incentive underneath it hasn't been thought through yet.)
Sources
Tiffany, Maybe You Missed It, but the Internet 'Died' Five Years Ago (The Atlantic, August 2021) (https://www.theatlantic.com/technology/archive/2021/08/dead-internet-theory-wrong-but-feels-true/619937/)
Shumailov et al., AI models collapse when trained on recursively generated data (Nature, July 2024) (https://doi.org/10.1038/s41586-024-07566-y)
NewsGuard, AI Tracking Center: AI content-farm sites (ongoing) (https://www.newsguardtech.com/special-reports/ai-tracking-center/)
Imperva/Thales, Bad Bot Report 2026: automated traffic above 53 percent of all web traffic in 2025 (April 2026) (https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age/)
Graphite, AI Now Writes as Many Online Articles as Humans (May 2026), superseding its own October 2025 study (https://graphite.io/five-percent/ai-now-writes-as-many-online-articles-as-humans-do)
C2PA, Coalition for Content Provenance and Authenticity: the Content Credentials standard (https://c2pa.org/)
Optional C. Reverse Engineering AI Products
Cursor, Claude Code, Lovable, Perplexity, NotebookLM case studies.
Take apart successful AI products to understand design, business, and architecture.
By the end of this chapter, you will be able to
Analyze an AI product's design choices through the lens of friction and guardrails
Infer a product's business model and assess its defensibility (moat)
Infer likely technical architecture from a product's observable behavior
C.1 Product design lens
The fastest way to learn product design is to take apart products that clearly nailed it. Pick a few that earned real adoption (Cursor [1], Claude Code [2], Lovable [5], Perplexity [3], NotebookLM [4]) and ask first what makes each one feel good to use, since 'feel' is usually deliberate engineering, not luck. Where does it remove friction compared to whatever it replaced, and what was the single insight that made people switch?
Then look at the constraints, because good AI products are defined as much by what they prevent as by what they enable. A model is powerful and unreliable (Chapter 9), leaving the design to keep users from getting lost or making a costly mistake. Perplexity forces citations [3], coding tools show diffs before applying them, and NotebookLM grounds answers in your uploaded sources [4], each a deliberate guardrail dressed up as a feature.
The deeper question to carry through every case study is how the product handles being wrong, because that's where AI UX is won or lost, and 12.5 lists what to look for. Reverse-engineering the design means noticing the dozens of small decisions behind the impressive demo, the ones that make the product trustworthy enough to use on real work.
How to reverse-engineer a product you didn't build: observe, guess, probe, and rebuild your way to a mental model.
What you can infer about a product's stack just from its UI and network traffic.
C.2 Business model lens
A product's business model quietly shapes every decision it makes, so identifying how each one actually earns money is half of understanding it. The common shapes are subscription (flat monthly fee), usage-based pricing (pay per token or per action, mirroring the provider's own API costs), and freemium (a free tier that converts some users to paid), each creating different pressures on what the product becomes.
Trace how the model bends the product. Usage-based pricing pushes a company to make heavy use feel worth it but also to control its own costs, since every customer action has a real token bill behind it (Chapter 13). Freemium forces a hard line (what's valuable enough to gate behind payment versus give away to drive adoption), and that line tells you what the company believes its real value is, making the pricing page a strategy document if you read it closely.
Finally, ask the question from the Feature Graveyard (12.6): what's this product's moat? That section covers what makes a wrapper vulnerable and a moat durable. Spotting where a successful product's defensibility actually lives is one of the most transferable lessons you can take from a case study into your own building.
C.3 Technical architecture lens
You won't have the source code, but you can infer a surprising amount of a product's architecture from how it behaves, and doing so turns the abstract concepts from earlier chapters into recognizable patterns in the wild. Watch the seams: does it stream its responses token by token (Chapter 7)? Does it answer from a fixed set of documents in a way that suggests retrieval (Chapter 4's RAG)? Does it take actions that imply tool calls behind the scenes?
Probe deliberately with edge cases, the way you'd test any system. Ask something just outside its knowledge and see whether it admits ignorance, hallucinates, or goes and fetches, since each behavior points at a different design. Then give it a very long input and watch how it handles context limits. The places a product strains or degrades are exactly where its architecture is showing through.
The payoff of this lens is confidence: once you can look at an impressive AI product and sketch a plausible version of how it's built (model plus retrieval plus tools plus guardrails), these products stop feeling like magic and start feeling like assembly. That demystification is the real goal of the whole course, and reverse-engineering the ones that work is one of the fastest ways to reach it.
Summary
You can learn a lot about an AI product without its source code. We read products through three lenses: design, business, and architecture. Small guardrail decisions reveal trustworthiness; the moat reveals the business; observable behavior like streaming, retrieval, and tool calls reveals the architecture. Good products are assembled deliberately, not by magic.
Key terms
Case study. A named, real product (Cursor, Claude Code, Lovable, Perplexity, NotebookLM) analyzed through the design, business, and architecture lenses to make abstract concepts recognizable in the wild.
Product design. The lens that asks what makes a product feel good to use and what it deliberately prevents: guardrails like forced citations or diffs shown before applying, dressed up as features.
Business model. The way a product makes money (subscription, usage-based, freemium) and the pressures that choice puts on product decisions.
Usage-based pricing. Charging in proportion to consumption (e.g. per API token or per request), common for AI products.
Moat. A durable advantage (data, network effects, switching costs) that protects a product from competitors.
Architecture. The technical design (model, retrieval, tools, guardrails) inferred not from source code but from observable behavior, such as whether a product streams responses, appears to retrieve from fixed documents, or takes actions implying tool calls.
Wrapper. A product that mostly packages another model provider's API with thin added value, versus a defensible platform around it.
Check your understanding
Q1. When reverse engineering a product's architecture without source access, what's the most reliable method?
Guessing at random and checking nothing
Assuming every AI product shares an identical architecture
Architecture can't be inferred without source code
Observing behavior: streaming, latency, what it retrieves or calls
Answer: D. Behavioral signals (response timing, what context it seems to draw on, tool-like actions) reveal a surprising amount about the underlying architecture.
Q2. According to the product design lens, what do guardrails like Perplexity's forced citations or a coding tool's diff-before-apply actually do?
Deliberate constraints, dressed as features, that stop costly mistakes
They exist to slow the user down and serve no functional purpose
They're legally mandated disclosures with no design purpose
They appear only in the free tier of any given product
Answer: A. Good AI products are defined as much by what they prevent as by what they enable. A guardrail is a considered response to the model being powerful and unreliable, not an accident.
Q3. What does a product's pricing model (subscription, usage-based, freemium) reveal, according to the business model lens?
Nothing useful, since pricing is essentially arbitrary
It shapes real pressures on what a company builds and gates
All AI products eventually converge on the same pricing
Pricing only matters for products sold to enterprises
Answer: B. The pricing page is effectively a strategy document: it exposes what the company believes is valuable enough to charge for and how it manages its own underlying costs.
Practice
Exercise 1. Pick one AI product you use regularly. Write one paragraph each on its product design, business model, and likely architecture, based only on what you can observe as a user. (Hint: Try a few edge-case inputs (very long, very short, ambiguous) and watch how the product handles them. That often reveals architectural choices.)
Exercise 2. Look at the pricing page of one AI product (subscription, usage-based, or freemium). Identify what's gated behind payment versus free, and write one sentence on what that line reveals about what the company believes its real value is. Then name its likely moat, or argue it doesn't have one. (Hint: If the product is mostly a thin wrapper around a single model API, ask the Feature Graveyard question from 12.6: would it survive the provider shipping this feature themselves?)
Sources
Cursor docs: architecture overview and how the editor talks to models (https://cursor.com/docs/agent/overview)
Claude Code docs: architecture and how the agent reads, edits, and runs code (https://code.claude.com/docs/en/overview)
Perplexity Help Center, How does Perplexity work: search, retrieval, and cited answers (https://www.perplexity.ai/help-center/en/articles/10352895-how-does-perplexity-work)
Google, NotebookLM: source-grounded answers and citations (https://blog.google/innovation-and-ai/technology/ai/notebooklm-google-ai/)
Lovable docs: how prompts become deployed full-stack apps (https://docs.lovable.dev/introduction/welcome)
Optional D. AI and Education
Learning with AI, integrity, tutors, assessment redesign.
How AI changes learning itself, including this very course.
By the end of this chapter, you will be able to
Explain the productive-struggle paradox in AI-assisted learning
Compare AI's effect on academic integrity to historical precedents like the calculator
Design an assessment that measures judgment rather than the recall a model could supply
D.1 Learning with AI tutors
An AI tutor has properties no human teacher can match at scale: infinitely patient, available at 2am, willing to re-explain the same concept five different ways until one lands, and able to adapt to one student's exact pace. For a motivated learner, this is genuinely transformative, the personalized tutoring that decades of research have found to produce substantial gains over ordinary classroom instruction [1][8], suddenly available to anyone. A 2025 trial found students learning more from a well-designed AI tutor than from in-class active learning [2].
But there's a catch that determines whether the tutor helps or hurts: learning happens in the struggle, not in the answer. The cognitive effort of working through a hard idea is what builds understanding, and an AI that hands over the finished answer removes exactly the friction that does the learning. The same tool can deepen understanding or short-circuit it, and which one depends entirely on how the student uses it. In one study, students given an unrestricted assistant performed worse on later unaided tests than students who never had it, while a version built with tutoring guardrails didn't cause the same harm [3].
This is the productive-struggle paradox, and it's the central tension of AI in education. A tutor prompted to 'explain why my approach is wrong' teaches. A tutor prompted to 'just give me the answer' replaces the thinking. The skill students now have to learn, and educators have to teach, is using AI to amplify their own thinking instead of substituting for it, which is the same calibrated-use judgment the whole course is about, applied to learning itself.
The same tool, two outcomes. Whether AI builds understanding or erodes it depends entirely on how you use it.
Where AI helps and where the learning has to be yours, mapped onto Bloom's taxonomy.
D.2 Academic integrity in an AI world
When a model can produce a passable homework answer in seconds, 'did you do this yourself' stops being a simple yes/no and becomes genuinely hard to define, let alone detect. AI-detection tools are unreliable enough that acting on them risks falsely accusing honest students [4], and they misfire disproportionately on writers whose first language isn't English [5], so the old enforcement model is quietly breaking down whether or not institutions admit it.
The more durable response, and the one many institutions are moving toward, is to stop treating AI as binary contraband and start defining what assisted-but-honest work looks like [7]. The useful precedent is the calculator: math education didn't collapse when calculators arrived, and the research consensus is that access generally helped achievement [6]. It adapted by deciding when the tool is appropriate, teaching the underlying concepts anyway, and assessing in ways the tool couldn't simply shortcut. Citation norms are a similar model of disclosed, governed use rather than prohibition.
This reframes integrity from 'did you avoid the tool' to 'did you actually learn, and were you honest about how.' That's harder to police but closer to the real goal, and it pushes the hard problem onto assessment design, because if your assessment can be fully completed by a model with no learning involved, the integrity problem is partly a symptom of the assessment, which is where the next section goes.
D.3 Redesigning assessment
An assessment that can be solved completely by pasting the prompt into a model has stopped measuring learning and started measuring access to a model, which every student now has. Rather than fighting that with bans and detectors, the more honest move is to redesign assessment so it measures what AI can't simply supply: judgment, process, and explanation instead of pure recall.
Several formats hold up better in this world. Live explanation, having a student talk through their reasoning in real time, is hard to fake because understanding can't be copy-pasted out loud. Process artifacts, like the chat history showing how a student arrived at an answer, make the thinking itself the thing being graded. And open-AI exams explicitly permit the tool and then test the harder skill: can you judge, correct, and build on AI output rather than just produce it?
Notice that this is exactly what this book assesses. Its labs and exercises reward planning, verifying, and judgment about AI output over memorized syntax, which isn't a coincidence so much as the thesis. The redesign that AI forces on education turns out to align with what actually matters in an AI-saturated working world, which is the optimistic reading of an otherwise disruptive shift: the assessments that survive AI are also the ones that were always measuring the right thing.
Summary
AI is reshaping how we learn, including this book. We covered tutoring that helps only when the student still does the cognitive work, integrity that's redefined and not abolished, and assessment formats that survive AI by measuring judgment instead of recall. This book is the example, with labs that reward planning and verifying over syntax recall.
Key terms
AI tutor. An AI system acting as a patient, always-available personal tutor, helpful only if the student still does the cognitive work.
Academic integrity. Honest authorship of one's work, redefined, not abolished, when AI assistance is available.
Assessment redesign. Shifting assessment from recall (which a model can supply) toward judgment, process, and explanation, using formats like live explanation, process artifacts, and open-AI exams.
Future classrooms. Classrooms shaped by personalized AI tutoring, disclosed AI use instead of prohibited, and assessments redesigned around judgment, following the calculator's precedent for adapting to a new tool rather than banning it.
Process artifact. Evidence of how a student arrived at an answer (drafts, chat history) used to assess genuine engagement.
Open-AI exam. An assessment that permits AI use and tests judgment about output rather than the recall a model could supply.
Personalized learning. Adapting pace and explanation to the individual learner, something AI tutors make feasible at scale.
Check your understanding
Q1. Why does an AI tutor only help learning under certain conditions?
AI tutors don't help learning under any conditions
AI tutors are only useful for already-advanced students
Learning needs the student to do the cognitive work, not collect answers
There are no conditions, since tutoring always helps equally
Answer: C. Getting an answer and understanding how you'd derive it are different outcomes. The tutor helps the second only if the student engages with the reasoning, not just the result.
Q2. What precedent does this chapter use to argue that AI won't collapse academic integrity the way many fear?
The invention and spread of the printing press
There's no precedent; this is an entirely unprecedented situation
The introduction of spell-check in word processors
The calculator, which math education absorbed by adapting its assessment
Answer: D. Math education didn't collapse when calculators arrived; it adapted its teaching and assessment. The chapter argues AI calls for the same kind of deliberate adaptation rather than prohibition.
Q3. Why does the chapter say an assessment that can be fully solved by pasting the question into a model has 'stopped measuring learning'?
Because it measures access to a model, which every student now has
Because assessments of that kind are now formally prohibited
Because models reliably refuse to answer assessment questions
Because pasting a question into a model is too slow to be practical
Answer: A. If a model can fully complete the assessment with no learning involved, the assessment is testing who has access to AI, not who understands the material, which pushes the fix toward redesigning what's being measured.
Practice
Exercise 1. Design one assessment question for this book's Chapter 9 (AI Slop) that couldn't be fully answered by pasting the question into a model with no further thought from the student. (Hint: Questions that require judgment about a specific, novel piece of output, not recall of a definition, are much harder to answer by copy-paste alone.)
Exercise 2. Pick a concept you're currently struggling with. Write two different prompts to an AI tutor about it: one that asks it to just give you the answer, and one that asks it to explain why your current approach is wrong or guide you toward the answer. Compare what you actually learned from each. (Hint: The productive-struggle paradox predicts the second prompt teaches more even though it's slower. Notice specifically where the first prompt let you stop thinking.)
Sources
Bloom, The 2 Sigma Problem (Educational Researcher, 1984) (https://doi.org/10.3102/0013189X013006004)
Kestin et al., AI tutoring outperforms in-class active learning (Scientific Reports, 2025) (https://doi.org/10.1038/s41598-025-97652-6)
Bastani et al., Generative AI without guardrails can harm learning (PNAS, 2025) (https://doi.org/10.1073/pnas.2422633122)
Weber-Wulff et al., Testing of detection tools for AI-generated text (Int. Journal for Educational Integrity, 2023) (https://doi.org/10.1007/s40979-023-00146-z)
Liang et al., GPT detectors are biased against non-native English writers (Patterns, 2023) (https://doi.org/10.1016/j.patter.2023.100779)
Hembree and Dessart, Effects of hand-held calculators in precollege mathematics education (JRME, 1986) (https://doi.org/10.5951/jresematheduc.17.2.0083)
Russell Group, Principles on the use of generative AI tools in education (July 2023), reproduced in full by King's College London, a signatory, because the Russell Group's own site blocks access (https://www.kcl.ac.uk/about/strategy/learning-and-teaching/ai-guidance/macro-level)
Nickow et al., The promise of tutoring for PreK-12 learning: a systematic review and meta-analysis of the experimental evidence (American Educational Research Journal, 2024) (https://doi.org/10.3102/00028312231208687)
Optional E. Vibe Debugging
Fix broken, poorly generated codebases with bad docs.
Students inherit messy AI-generated projects and learn to debug, reverse engineer, and pay down technical debt.
By the end of this chapter, you will be able to
Apply systematic fault isolation to a bug in an unfamiliar codebase
Infer the intent behind undocumented code before modifying or deleting it
Identify and consolidate AI-generated technical debt without a full rewrite
E.1 Systematic debugging
This chapter drops you into the real-world situation vibecoding creates constantly: a codebase that mostly works, that you didn't write, with sparse or wrong documentation and a bug somewhere inside. The wrong instinct is to read the whole thing top to bottom hoping the problem jumps out. In any non-trivial codebase, it won't. Debugging is a method, not a staring contest [1][2].
The method starts with reproduction: a reliable set of steps that triggers the bug every time. A bug you can reproduce on demand is a bug you can corner. One that appears randomly is one you'll chase for days (which is why Chapter 10's reproducibility work pays off here). Only once you can summon the failure at will should you start narrowing where it lives.
Then you narrow systematically rather than randomly. Bisect the code path: check a value halfway through, and you've instantly halved the search space. Add logging at the boundaries between components to see where good input becomes bad output. Form one specific hypothesis ('I think this function returns null when the list is empty') and test exactly that [4]. This is the same discipline as Chapter 10, now with no original author to ask, and AI is a genuine ally for it, since 'here's the error and the relevant code, what are the likely causes' is one of the things models are most useful for.
What you inherit when an app builder hands you a working product: capability you didn't write and don't yet understand.
Debugging code you didn't write adds one step, reading, to the front of the usual loop.
E.2 Reverse engineering intent
AI-generated code has a distinctive property: it often does something for a reason that exists nowhere in the code or comments, because the model that wrote it had a reason in the moment and no memory of it afterward. So you'll find a branch, a check, or a transformation with no explanation, and the work of changing it safely is recovering the intent that's no longer written down anywhere.
The technique is to read the code as evidence of the requirements it was responding to. A weird-looking null check is evidence that some input really did arrive null at some point. An odd special case is evidence that some real situation needed special handling. The code is a fossil record of problems the author hit, and 'this looks unnecessary, I'll delete it' is exactly how you reintroduce a bug someone already fixed, because Chekhov's null check is on the wall for a reason.
So before removing anything that looks pointless, generate hypotheses for why it exists and check them, rather than assuming it's dead code. That caution is the difference between improving a codebase and breaking it in a way that only shows up in production weeks later. Reverse engineering intent is detective work, where absence of an explanation isn't evidence of absence of a reason.
E.3 Paying down technical debt
Technical debt, the metaphor for the future cost of shipping code that reflects only your current, partial understanding of the problem [5], has a recognizable shape when it comes from rapid AI-assisted generation, and it's usually not dramatic broken code so much as inconsistency. Because a model generates each piece fresh with no enforced memory of its earlier choices (the same root cause as the inconsistent UI in Chapter 5), you get three different patterns for the same operation, copy-pasted logic that has since drifted apart, and abstractions introduced once and never reused, where each piece is fine on its own and the whole is a maze.
This kind of debt is dangerous precisely because it's quiet. It doesn't crash. It just makes every future change slower and riskier, because you can't trust that fixing something in one place fixes it everywhere, and you can't predict which of the three patterns a given feature used. Left alone, it compounds, until a codebase that 'works' becomes one nobody can safely change, the AI-era version of an unmaintainable legacy system, reached in months instead of years.
Paying it down is consolidation, not a rewrite. The instinct to throw it all out and start clean is usually a trap (you'd lose all the embedded intent from the last section and likely reintroduce old bugs). The disciplined move is to pick one pattern deliberately and migrate the others toward it, one safe refactor at a time, verified by tests at each step (Chapter 3) [3]. Done steadily, this is how an inherited mess becomes a codebase you actually own, which is the entire skill this chapter exists to build.
Summary
Inheriting code you didn't write (increasingly code an AI wrote) is now a normal starting point. We covered systematic debugging by narrowing the search and reverse-engineering missing intent. We recognized technical debt that shows up as quiet inconsistency, not dramatic breakage. The safe fix: consolidate toward one pattern, verified by tests. Don't rewrite from scratch.
Key terms
Fault isolation. Narrowing down where a bug actually lives, rather than reading everything and hoping it jumps out.
Reproduction (repro). A reliable set of steps that triggers a bug every time, the prerequisite for debugging it.
Bisection. Repeatedly halving the search space (of code or commits) to locate the source of a bug or regression.
Reverse engineering. Recovering the intent behind undocumented AI-generated code by reading it as evidence of the requirement it was responding to, since the model that wrote it had a reason in the moment and no memory of it afterward.
Stack trace. The chain of function calls leading to an error, a primary clue for locating a fault.
Technical debt. Accumulated shortcuts and inconsistency that make future changes harder, often produced fast by unguided AI generation.
Check your understanding
Q1. What's the recommended first step when debugging unfamiliar, undocumented code?
Rewrite the whole file immediately from scratch
Reproduce it reliably, then narrow with a specific hypothesis
Delete whatever code looks suspicious on a first read
Assume the bug sits in the most recently changed file
Answer: B. Reliable reproduction plus systematic narrowing beats reading-and-hoping, especially in code you didn't write.
Q2. Why does this chapter warn against deleting a weird-looking, unexplained check in AI-generated code?
Because deleting any code will reliably break the build
Because AI-generated code can't legally be edited by hand
Because it's likely evidence of a real case the code had to handle
Because unexplained code is always deliberate and correct
Answer: C. The chapter frames code as a fossil record of problems the author (even an AI) hit. Absence of an explanation isn't evidence of absence of a reason, so hypotheses should be checked before removal.
Q3. What shape does AI-generated technical debt usually take, according to this chapter?
Dramatic code that crashes immediately on its very first run
Missing semicolons and other straightforward syntax errors
Debt appears only in code by junior developers, never AI
Quiet inconsistency: many patterns for one job, drifted duplicates
Answer: D. Because a model generates each piece fresh with no enforced memory of earlier choices, the debt shows up as inconsistency instead of breakage, which makes it dangerous precisely because it doesn't announce itself.
Practice
Exercise 1. You're handed a function with a weird-looking null check on one specific field. Without asking anyone, write down two hypotheses for why that check might exist before deciding whether to remove it. (Hint: A defensive-looking check that seems unnecessary today may be guarding against an input shape from an earlier version of the code or an edge case you haven't hit yet.)
Exercise 2. Take a bug you can reliably reproduce in a real or practice codebase. Instead of guessing at the fix, write down one specific hypothesis about where it lives, then bisect toward it: add a log or check a value halfway through the code path before touching anything else. (Hint: Resist the urge to fix on sight. The goal here is practicing the narrowing method itself, not landing the fix fast.)
Sources
Zeller, Why Programs Fail: A Guide to Systematic Debugging (2009) (https://www.whyprogramsfail.com/)
Agans, Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems (2002) (https://debuggingrules.com/)
Fowler, Refactoring: Improving the Design of Existing Code (martinfowler.com) (https://martinfowler.com/books/refactoring.html)
Google SRE Book, Effective Troubleshooting: hypothesis-driven fault isolation (https://sre.google/sre-book/effective-troubleshooting/)
Ward Cunningham, The WyCash Portfolio Management System: the original technical debt metaphor (OOPSLA, 1992) (https://c2.com/doc/oopsla92.html)
Learn from real, public AI disasters so you don't repeat them.
By the end of this chapter, you will be able to
Analyze a real AI failure case study to find its root cause
Connect public AI incidents to specific verification or oversight steps that were skipped
Explain why most AI disasters are process failures, not model failures
F.1 Hallucinated and misleading
The most instructive AI failures are the ones that played out in public with real consequences, and the canonical example is the lawyers (multiple, in multiple jurisdictions) sanctioned for filing legal briefs full of case citations that an AI invented and that simply don't exist. A New York court sanctioned two attorneys over fabricated precedent in 2023 [1], and the English High Court dealt with the same problem in 2025 [2]. Real attorneys, in a profession where checking your citations was always the entire job, submitted fabricated precedent to a court because the model produced it confidently and they didn't verify.
It's the perfect cautionary tale because it's 'looks right' syndrome (Chapter 9) with a courtroom attached. The fake citations had real-sounding case names, plausible reporter numbers, the exact format of genuine ones, everything except existence. Nothing on the surface warned the lawyer, which is the whole point: the more authoritative AI output looks, the more it earns an unearned trust, and a legal brief is about as authoritative as text gets.
Generalize past law, and the pattern is everywhere: fabricated statistics in reports, invented quotes in articles, made-up API methods in code. Any domain where being wrong is costly is a domain where unverified AI output is a liability, and the people most embarrassed were professionals who simply trusted fluency over verification one time too many.
A hall of fame of expensive mistakes. Every one has happened to someone who trusted output they didn't read.
Not all bugs are equal. The point of verification is to keep failures on the left end of this bar.
F.2 Failed products and security incidents
Beyond individual mistakes are the product-level disasters. The recurring shape is an AI product shipped without enough testing against adversarial or edge-case input, then pulled shortly after launch once real users found the offensive, false, or embarrassing output it would happily produce at scale. Microsoft's Tay chatbot lasted about a day in 2016 [3], and Meta's Galactica lasted three days in 2022 [4], though in each case the demo had worked. The world isn't a demo, and the gap between the two is where these products died.
The security incidents are quieter but more dangerous. AI-connected systems have leaked data through insufficient input validation, through authorization gaps that let one user reach another's data (the exact failure from Chapter 6, and the shape of a 2023 ChatGPT bug that exposed other users' payment details [5]), and increasingly through prompt injection (Chapter 14) [8], an attacker hiding instructions in content an AI agent reads, hijacking it into exfiltrating data or taking actions it never should have. None of this is exotic, since these are textbook vulnerabilities met by textbook neglect.
What makes these worth studying as case studies and not headlines is that each has a traceable root cause, and the root cause is almost never 'the AI was too dumb.' It's a specific skipped safeguard: a missing validation, an untested edge case, an agent given more power than oversight. An AI coding tool deleted a live production database during a code freeze in 2025 [6], and a Canadian tribunal held an airline responsible for a refund policy its chatbot invented [7]. Reading the post-mortems trains you to see the safeguard before the incident, which is the only time seeing it is cheap.
F.3 Lessons for builders
Stack these cases up and the failure is rarely that the model was bad. A verification, testing, or oversight step, the kind this book treats as mandatory in Chapters 9, 10, and 14, was skipped or quietly downgraded to optional under deadline pressure. The steps are the unglamorous ones the core chapters insist on: verify against a source, test the edge cases, red team your own system, keep a human in the loop when the stakes are high. The people in this hall of fame aren't a different species of careless, only builders who skipped a step you now know not to skip, which is exactly why the discipline is learnable and worth learning. That choice, made consistently, is the entire difference between shipping something trustworthy and becoming a case study.
Summary
Real AI failures teach faster than warnings. We looked at experts caught trusting fluency over verification and products shipped without testing adversarial input. We covered quieter security incidents from validation gaps and prompt injection. The pattern is clear: a verification habit, not expertise, is what immunizes you. The demo working isn't the world working.
Key terms
Case study. A real, public AI failure (a sanctioned legal filing, a pulled product, a data leak) analyzed for its traceable root cause, treated as instruction rather than headline.
Security incident. A breach caused by insufficient input validation, an authorization gap letting one user reach another's data, or prompt injection hijacking an agent into exfiltrating data or taking unauthorized actions.
Post-mortem. A blameless write-up after a failure, analyzing what happened and how to prevent a recurrence.
Root cause. The underlying reason a failure occurred, as opposed to its surface symptom.
Cherry-picked demo. A demonstration showing only the best-case results, hiding how often the system actually fails.
Misleading demo. A demo that implies a capability the product doesn't reliably have, a recurring source of public AI embarrassment.
Check your understanding
Q1. What's the common root cause behind most public AI failure case studies, per this chapter?
A verification or oversight step was skipped under deadline pressure
The underlying models are fundamentally broken in some way
AI products of this kind always fail eventually
Public failures are random and have no common cause
Answer: A. Most disasters trace back to a missed verification or oversight step, not an unavoidable flaw in the technology itself.
Q2. What made the fabricated legal-citation cases so dangerous, according to this chapter?
The citations were obviously fake to anyone who checked them
They had real-sounding names and the exact format of genuine ones
The lawyers involved had no formal legal training at all
Courts don't check the citations included in legal briefs
Answer: B. This is 'looks right' syndrome with a courtroom attached: the more authoritative AI output looks, the more it earns unearned trust, and a legal citation is about as authoritative as text gets.
Q3. What do the quieter security incidents in this chapter (data leaks, authorization gaps, prompt injection) have in common?
They're exotic vulnerabilities found only in AI systems
They only occur at companies with no security team at all
They're textbook vulnerabilities met by textbook neglect
They can't be prevented even by careful engineering work
Answer: C. The chapter is explicit that these aren't exotic: they're the same validation and authorization failures from earlier chapters, just showing up faster because AI-assisted speed generates new attack surface quickly.
Practice
Exercise 1. Find one publicly reported AI failure (legal, product, or security). Identify which specific chapter's practice (verification, testing, oversight, injection defense) would have caught it if followed. (Hint: Search news coverage rather than social media threads for a more reliable account of what actually happened.)
Exercise 2. Take the fabricated legal-citation cases from F.1. Write a one-sentence policy a law firm could adopt that would have caught the fake citations before filing, and explain why it works even though the citations looked completely authentic. (Hint: The fix isn't 'read more carefully'. It's a concrete, mechanical check, like requiring every citation to be pulled up in an actual case database before filing.)
Sources
Mata v. Avianca, Opinion and Order on Sanctions, Castel, J. (S.D.N.Y., June 22, 2023) (https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.54.0_1.pdf)
Ayinde v. Haringey and Al-Haroun v. QNB, fake AI citations before the High Court, [2025] EWHC 1383 (Admin) (June 2025) (https://caselaw.nationalarchives.gov.uk/ewhc/admin/2025/1383)
Lee (Microsoft), Learning from Tay's introduction (Official Microsoft Blog, March 2016) (https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/)
Heaven, Why Meta's latest large language model survived only three days online (MIT Technology Review, November 2022) (https://www.technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science/)
Abrams, OpenAI: ChatGPT payment-data leak caused by open-source bug (BleepingComputer, March 2023) (https://www.bleepingcomputer.com/news/security/openai-chatgpt-payment-data-leak-caused-by-open-source-bug/)
Nolan, AI coding tool wiped a live database: the Replit/SaaStr incident (Fortune, July 2025) (https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/)
McCarthy Tetrault, Moffatt v. Air Canada: a misrepresentation by an AI chatbot (February 2024) (https://www.mccarthy.ca/en/insights/blogs/techlex/moffatt-v-air-canada-misrepresentation-ai-chatbot)
Greshake et al., Not what you've signed up for: indirect prompt injection in LLM-integrated applications (AISec workshop at ACM CCS, 2023) (https://doi.org/10.1145/3605764.3623985)
Use AI to explore data without abandoning statistical rigor.
By the end of this chapter, you will be able to
Use AI to accelerate exploratory data analysis while still framing the right questions
Distinguish correlation from causation and check a finding against held-out data
Recognize overfitting and misleading visualization in AI-assisted analysis
G.1 Exploratory analysis and visualization
Exploratory data analysis (EDA), the early phase of poking at a dataset to understand its shape before any formal conclusions, is one of the places AI assistance shines brightest. Describe a dataset in plain language, and a model will generate summary statistics, flag which variables might relate to each other, and produce charts without you hand-writing a line of plotting code. Work that used to take an afternoon of fiddling with a charting library happens in seconds.
But that speed relocates the hard part rather than removing it. When generating an analysis is nearly free, the scarce skill becomes deciding which questions are worth asking, because the model will just as happily produce a beautiful chart for a meaningless question as for a crucial one, having no sense of what matters in your domain, so that judgment stays with you and is now most of the job.
So the productive pattern is to let AI handle the mechanics while you drive the inquiry. Use it to rapidly test the questions you think matter, to surface things you wouldn't have thought to chart, and to get unstuck, but stay the one steering toward what's actually meaningful. EDA with AI is a conversation where the model does the typing and you do the wondering, and the quality of the wondering is what determines whether you learn anything.
Load โ Clean โ Explore โ Model โ Communicate
The data-science pipeline. AI accelerates every stage, but the judgment at each one stays yours.
A clean division of labor for AI-assisted analysis: let the model do the typing, keep the thinking.
G.2 Keeping statistical reasoning honest
A model will compute a correlation, run a test, or fit a curve instantly and present the result with total fluency, and that fluency is exactly the trap, because statistical rigor lives in the questions a number can't answer on its own, so the model gives you the what and no more, leaving the question of whether it means anything to you.
Three questions guard against the most common errors. Does this correlation imply causation? Almost never on its own. Two things moving together may share a hidden cause (a confounder), and a model stating the correlation confidently doesn't change that. Is the sample large enough to support the conclusion being drawn? A striking pattern in twelve data points deserves far more skepticism than the same pattern in twelve thousand. And would the result survive being checked against a held-out portion of the data it wasn't found in?
This is the same calibrated-skepticism the whole book teaches, applied to numbers instead of code or prose. An AI-generated analysis is a draft conclusion, not a verified one, and the danger is that statistics dressed in confident output and a clean chart feel authoritative in a way that disarms the very scrutiny they most require. Honest analysis means treating the model's statistical claims exactly as provisionally as any other unverified AI output.
G.3 Common mistakes AI-assisted analysis invites
Two mistakes recur often enough to name. The first is overfitting a narrative to noise: finding a pattern in a small or noisy dataset and treating it as a real signal. AI makes this easier precisely because it lowers the cost of looking, and if you run enough analyses, some will turn up an impressive-looking pattern by pure chance. That's p-hacking, the accidental version, and the speed of AI assistance is exactly what makes it easy to do without realizing [1].
The second is letting presentation outrun substance. An AI-generated chart can be polished and visually convincing while implying a far stronger conclusion than the underlying data supports: a misleading axis, a cherry-picked range, a correlation drawn as if it were a trend. Polish isn't rigor (the same lesson as Chapter 9's 'looks right' syndrome), and a gorgeous chart is one of the most persuasive ways to be wrong.
The defense is discipline you impose, since the model won't: decide what you're testing before you go looking, be honest about sample size and uncertainty, and check a surprising finding against held-out data before you believe it. AI is a genuine accelerator for good data science and an equally good accelerator for bad data science. Which one you get depends entirely on whether you bring the statistical honesty the model can't supply.
Summary
AI accelerates data exploration without removing the need for rigor. We covered letting the model do the typing while you frame the questions, separating correlation from causation, and checking against held-out data. We spotted overfitting and misleading charts. A beautiful chart is a persuasive way to be wrong, so the same calibrated skepticism applies to numbers.
Key terms
EDA (Exploratory Data Analysis). The early phase of examining a dataset to understand its shape, distributions, and relationships before formal analysis.
Visualization. Charts that an AI tool generates from a plain-language description, powerful for exploration but just as capable of implying a stronger conclusion than the data supports through a misleading axis, cherry-picked range, or a correlation drawn as a trend.
Statistical reasoning. The judgment a computed number can't supply on its own: whether a correlation implies causation, whether the sample is large enough to support a conclusion, and whether a finding survives a check against held-out data.
Correlation vs. causation. Two variables moving together doesn't mean one causes the other. Confirming causation needs more than a computed correlation.
Confounder. A hidden third variable that influences two others, creating a misleading correlation between them.
Overfitting. Reading patterns into noise, a model or narrative that fits the sample but fails to generalize.
p-hacking. Trying many analyses until one looks 'significant,' producing false findings. AI tools make it easy to do accidentally.
Held-out data. Data deliberately set aside and not used during analysis or training, used to check whether a finding generalizes.
Check your understanding
Q1. Why is correlation found by an AI tool not automatically evidence of causation?
AI tools are unable to compute a correlation at all
Correlation and causation amount to the same claim
This applies to hand-computed correlations, not AI-generated ones
Correlation shows two variables move together, causation needs more
Answer: D. A computed correlation is just a number. Confirming causation requires additional reasoning about confounders and mechanism that the tool didn't do for you.
Q2. According to this chapter, what's the scarce skill once AI makes generating charts and summary statistics nearly free?
Deciding which questions are worth asking at all
Writing all of the plotting code by hand yourself
Formatting charts with a better color palette
Sourcing and downloading much larger datasets
Answer: A. AI has no sense of what matters in your domain. It handles the mechanics; framing which questions are actually meaningful stays entirely the analyst's job.
Q3. Why does this chapter call an AI-generated chart 'one of the most persuasive ways to be wrong'?
Because generated charts are always factually incorrect
Because a polished chart implies more than the data supports
Because AI can't generate usable charts at all
Because charts may not legally appear in reports
Answer: B. Polish isn't rigor. A gorgeous chart disarms the scrutiny it most needs, the same 'looks right' trap from Chapter 9 applied to data presentation.
Practice
Exercise 1. Take a small dataset and ask an AI tool to find 'interesting patterns.' For each pattern it reports, assess whether the sample size is large enough to trust the finding. (Hint: A pattern in a sample of 12 rows deserves much more skepticism than the same pattern in a sample of 12,000.)
Exercise 2. Before touching a dataset, write down three specific questions worth asking of it and rank them by 'would the answer actually change a decision.' Only then ask an AI tool to generate charts for the top-ranked question, and compare against what it would have produced unprompted for 'show me something interesting.' (Hint: The point is to feel the difference between AI-driven exploration and AI-supplied direction. Notice how many of the unprompted charts answer questions nobody needed answered.)
Sources
Simmons, Nelson and Simonsohn, False-Positive Psychology, the canonical p-hacking paper (Psychological Science, 2011) (https://doi.org/10.1177/0956797611417632)
Optional H. Open-Source and Local Models
Ollama, local inference, privacy, model selection.
Run capable models on your own hardware for privacy and control.
By the end of this chapter, you will be able to
Run an open-weights model locally with a tool like Ollama
Decide when local inference is justified by privacy or data-sensitivity requirements
Select a quantized model size appropriate to available hardware
H.1 Running models locally with Ollama
Everything so far has assumed you call a model over the internet through a hosted API. There's another option: download an open-weights model (one whose trained parameters are publicly available) and run inference directly on your own machine. Tools like Ollama make this nearly as easy as installing an app [1], turning your laptop into the thing serving the model. Those same tools now also offer hosted variants of larger models that run on the vendor's servers instead of yours, reached by adding a tag to the model name and signing in to an account [4]. The purely local path still needs no account, so confirm which of the two you're running before you count on the privacy properties described in the next section.
The trade-off is real and worth stating plainly. Local models are generally smaller and less capable than the frontier hosted models, because you're limited by your own hardware, not a data center's. In exchange, you get three things the API can't offer: no per-request cost (run it a million times for free), no dependency on a provider's uptime or pricing, and, the big one, nothing ever leaves your machine.
This makes local inference a genuinely different tool, not a worse version of the same one. For a high-volume, simple task where API costs would pile up, or for development where you want to iterate without metering every call, a capable local model can be the right choice precisely because the economics and privacy flip. Knowing the option exists expands what you can build, especially when the constraints aren't really about raw model quality.
The core trade-off in running models yourself: privacy and cost versus raw capability and scale.
Quantization shrinks a model to fit your hardware, trading a little quality for a lot less memory.
H.2 Why privacy drives local inference
The single most compelling reason to run locally is data that must not leave your control: health records, legal documents, proprietary source code, anything under a confidentiality obligation. For these, the relevant question from Chapter 14 ('does the provider retain or train on this?') has a clean answer if the data never reaches a provider at all. Local inference makes the privacy question disappear by removing the third party.
Often this is a hard requirement. Regulated industries and many enterprises simply can't send certain data to an external API regardless of how trustworthy the provider's stated policy is, because the rule is about where the data goes, not how it's handled once there. For those cases, a somewhat-less-capable model that runs entirely on-premises beats the best hosted model that's off the table for compliance reasons.
The honest framing is a trade you make deliberately: you accept generally weaker model quality in exchange for absolute control over the data. Whether that trade is worth it depends entirely on the task, overkill for a casual side project, non-negotiable for handling patient records. The skill is recognizing which situation you're in, instead of defaulting to the hosted API out of habit when the data is the kind that shouldn't travel.
H.3 Model selection for local hardware
Running locally turns model choice into a concrete hardware question, because the model has to fit in your machine's memory, specifically a GPU's VRAM for good speed. A model's size is roughly its parameter count, and bigger generally means more capable and more memory-hungry, so 'which model' becomes inseparable from 'what can my machine actually hold.'
Quantization is the technique that makes this practical. It compresses a model's weights to a lower numeric precision [2], dramatically shrinking its memory footprint at some cost to quality, the difference between a model that won't load and one that runs comfortably on a laptop. Most local-model work involves picking a quantized version sized to your hardware, and a well-chosen quantization often gives up surprisingly little for a large memory saving, with 4-bit precision a common sweet spot [3].
Selecting well means matching three things: the task's difficulty, the model's capability, and your hardware's limits. A simple classification or formatting task may run great on a small quantized model. A complex reasoning task may need more than your machine can hold, which is your signal to use a hosted API instead. This is the same model-selection judgment from Chapter 13 (use the smallest model that clears the bar), now constrained by the physical box in front of you rather than a billing dashboard.
Summary
Running capable models on your own hardware is a different tool, not a worse hosted API. We covered when local inference wins on cost or privacy, the deliberate trade of some quality for full data control, and how model choice becomes a hardware question once a model has to fit in memory. The skill: recognizing which situation you're in.
Key terms
Ollama. A tool that makes downloading and running open-weights models on your own machine nearly as easy as installing an app, turning your laptop into the thing serving the model. It also offers hosted variants that run on the vendor's servers instead, so check which kind you are running when privacy is the reason you went local.
Local inference. Running a model on your own hardware instead of calling a hosted API, with no per-request cost and nothing leaving your machine.
Open weights. A model whose trained parameters are publicly downloadable, letting anyone run or fine-tune it.
Parameters. The learned numeric weights of a model. More parameters generally mean more capability and more memory needed.
Quantization. Compressing a model's weights to lower precision so it fits in less memory, trading some quality for feasibility on consumer hardware.
VRAM. The memory on a GPU, the main hardware constraint on which local model size you can run.
Context length. The size of a local model's context window, which also consumes memory and varies by model.
Model selection. Matching task difficulty, model capability, and hardware limits when choosing a local model, the same 'smallest model that clears the bar' judgment from Chapter 13, now constrained by the physical machine rather than a billing dashboard.
Check your understanding
Q1. What does quantization trade off in a local model?
It trades away network speed to buy back disk space
It carries no meaningful trade-off in practice
It shrinks the weights to fit memory, costing some quality
It affects training only and never affects inference
Answer: C. Quantization is what makes running a large model on consumer hardware feasible, at the cost of some precision/quality.
Q2. According to this chapter, what's the single most compelling reason to run a model locally instead of through a hosted API?
Local models are reliably more capable than any hosted model
Local models require no meaningful setup at all
Hosted APIs cost more than local inference at absolutely any volume
Data that must not leave your control, since no third party is involved
Answer: D. For data under a confidentiality obligation, running locally makes the 'does the provider retain this?' question disappear by removing the provider from the picture entirely.
Q3. How does quantization make local inference practical?
It compresses the weights to lower precision, shrinking memory use
It raises the model's parameter count to improve its accuracy
It converts a local model into a hosted API for you
It changes nothing about memory usage, only raw speed
Answer: A. Since a local model has to fit in your machine's memory (GPU VRAM for speed), quantization trades some precision for a large reduction in memory footprint, often for surprisingly little quality loss.
Practice
Exercise 1. Install Ollama and run one small open-weights model locally. Compare one response from it to the same prompt sent to a hosted model. Note the capability gap you observe. (Hint: Try a task that needs broad world knowledge versus a task that's mostly pattern completion. The gap often shows up more on the former.)
Exercise 2. Look up the specs (VRAM/RAM) of a laptop or machine you have access to. Then find two quantized versions of the same open-weights model sized differently, and decide which one that machine could realistically run. Name one task you'd trust the smaller quantization for and one you wouldn't. (Hint: If you don't have GPU specs handy, use a typical student laptop's specs as a stand-in. The point is matching model size to a real memory budget, not owning specific hardware.)
Sources
Ollama: run open models locally (https://ollama.com)
Hugging Face Transformers docs: quantization overview (https://huggingface.co/docs/transformers/en/quantization/overview)
Dettmers and Zettlemoyer, The case for 4-bit precision: k-bit inference scaling laws (arXiv 2212.09720, 2022) (https://doi.org/10.48550/arXiv.2212.09720)
Ollama docs: cloud models, which run on Ollama's servers and need a signed-in account or an API key (https://docs.ollama.com/cloud)
Optional I. Fine-Tuning and Custom Models
Fine-tuning, LoRA, synthetic data, evaluation.
Specialize a model for your task when prompting isn't enough.
By the end of this chapter, you will be able to
Explain when fine-tuning is warranted compared to prompting alone
Compare full fine-tuning to parameter-efficient methods like LoRA
Evaluate a fine-tuned model against a held-out set rather than by inspection alone
I.1 Fine-tuning and LoRA
Most of this book gets a model to behave through prompting and context, the cheap, fast, reversible levers. Fine-tuning is the heavier tool you reach for when those aren't enough: it actually adjusts a pre-trained model's weights on your own examples so it specializes toward a specific task, format, or style. The important judgment is knowing this is a last resort, not a first one, because prompting solves more than beginners expect.
When you do fine-tune, the method matters for cost. Full fine-tuning updates every weight in the model and is expensive in compute and storage, largely the domain of well-resourced labs. LoRA (Low-Rank Adaptation) [1] and the broader family of parameter-efficient methods (PEFT) [2] instead train a small set of added parameters while leaving the original model frozen, capturing most of the benefit at a fraction of the cost. That efficiency is why LoRA, not full fine-tuning, is the practical default for almost everyone outside the big labs.
Fine-tuning also carries risks that prompting doesn't, which is part of why it's a later resort. Push too hard on a narrow task, and the model can suffer catastrophic forgetting, getting better at your examples while losing broader abilities it used to have [3]. And unlike a prompt you can edit in a second, a fine-tune is a training run you have to redo to change. The right mental model: fine-tune to bake in a stable behavior you've already validated you need, not to experiment.
Base model โ Dataset โ Train โ Evaluate โ Deploy
The fine-tuning pipeline. Most of the work is in the dataset and the evaluation, not the training run.
Ways to adapt a model to your needs, cheapest first. Climb only as high as the problem actually requires.
I.2 Synthetic data for training
Fine-tuning needs example data, and that's frequently the binding constraint. Real labeled examples are scarce, expensive to collect, or legally fraught to use. Synthetic data, examples generated by another model, has become a common way to fill the gap, and it can work genuinely well: you can produce thousands of varied training cases for a fraction of the cost of human labeling.
The catch is that synthetic data inherits the generating model's flaws. Whatever biases, blind spots, or subtle errors that model has can get baked into your examples and then trained into the new model, a quiet way to amplify a problem instead of solving it. There's also the model-collapse concern from Chapter 9 at small scale: train on a model's output, and you risk concentrating its quirks instead of learning from reality.
So synthetic training data deserves exactly the scrutiny the whole course demands of any AI output (Chapters 9 and 10): sample it, check it for the same problems you'd check generated code for, and verify it actually represents the real distribution you care about before you let it shape a model. Synthetic data is a powerful accelerator, but 'a model generated it' isn't 'it's correct,' and unverified bad data is worse here than usual, because its errors get permanently trained in.
I.3 Evaluating a fine-tuned model
The most important and most skipped step is proving the fine-tune actually helped, because fine-tuning isn't automatically an improvement. It can easily make a model better at your handful of examples and worse in ways you didn't measure, and eyeballing a few outputs that look nicer isn't evidence, which is the 'looks right' trap of Chapter 9 applied to a training run.
Real evaluation means benchmarking (Chapter 10): score the fine-tuned model against the original base model on a held-out set of examples it never saw during training, using a rubric or known-good answers, where the held-out part is non-negotiable. Testing on data the model trained on tells you it memorized, not that it generalized, which is the only thing you actually care about. A measurable win on unseen examples is what justifies shipping the fine-tune.
This closes the loop the whole chapter implies: fine-tuning is an experiment, and an experiment without a measurement is just a hope. Define how you'll know it worked before you start, then check honestly, and be willing to conclude it didn't help and revert to prompting, which happens more often than enthusiasts admit. Treating a fine-tune with the same evidence standard as any other change is what separates real specialization from an expensive way to feel productive.
Summary
Fine-tuning is a last resort, not a first move, since most 'we need to fine-tune' moments are really prompt problems. We covered full fine-tuning versus LoRA and the risks a prompt doesn't carry, like catastrophic forgetting. We covered the appeal and danger of synthetic data, which bakes its errors in permanently. Prove a change helped on held-out data, or revert to prompting.
Key terms
Fine-tuning. Adjusting a pre-trained model's weights on your own examples so it specializes toward a task or style.
LoRA (Low-Rank Adaptation). A lightweight fine-tuning method that trains a small set of added parameters, getting most of the benefit far more cheaply than full fine-tuning.
PEFT (Parameter-Efficient Fine-Tuning). The broader family of methods, including LoRA, that adapt a model by training only a small fraction of parameters.
Base vs. instruct model. A base model is the raw pre-trained model before task-specific adaptation; an instruct model has already been fine-tuned to follow instructions and hold a conversation. Evaluating your own fine-tune against the original base model on held-out data is how you prove the extra tuning actually helped.
Synthetic data. Training examples generated by another model to fill gaps when real labeled data is scarce, itself needing scrutiny.
Catastrophic forgetting. The loss of a model's broader abilities when fine-tuning on a narrow task degrades what it could previously do.
Held-out set. Examples withheld from training, used to honestly measure whether fine-tuning actually improved the model.
Check your understanding
Q1. What does LoRA offer compared to full fine-tuning?
Identical cost to full fine-tuning, with no added benefit
Most of the benefit at a fraction of the compute and storage cost
It's strictly worse than full fine-tuning in every case
It removes the need to supply any training data at all
Answer: B. LoRA trains a small set of additional parameters rather than the whole model, making specialization far cheaper.
Q2. Why does this chapter call fine-tuning 'a last resort, not a first move'?
Because fine-tuning is technically out of reach for most developers
Because fine-tuning reliably makes the base model worse
Because prompting solves more than expected, and fine-tuning carries risks
Because only large labs are legally permitted to fine-tune models
Answer: C. Most 'we need to fine-tune' moments are really prompt problems. Fine-tuning is heavier and riskier, so it's reserved for when the cheap, reversible levers of prompting and context genuinely aren't enough.
Q3. Why is synthetic training data riskier to leave unverified than most other AI output, per this chapter?
Synthetic data is reliably higher quality than real collected data
Synthetic data can't be checked for quality by any means
It isn't actually any riskier than other unverified AI output
The generating model's errors get baked into the new model permanently
Answer: D. A bad piece of generated code is a bug you can fix later. Bad synthetic training data gets trained into the model's weights, quietly amplifying the generating model's flaws rather than solving anything.
Practice
Exercise 1. Design an evaluation plan (not the actual fine-tuning) for a model you wanted to specialize for customer-support replies. Define the held-out test set and the metric you'd use to decide if tuning helped. (Hint: Make sure the held-out examples were never used during tuning. Testing on training data tells you nothing about generalization.)
Exercise 2. Take a task you think might need fine-tuning. Argue first for why better prompting or context alone could solve it, then argue for why it might genuinely need fine-tuning instead. If you land on fine-tuning, state whether LoRA or full fine-tuning fits, and why. (Hint: Most 'we need to fine-tune' moments turn out to be prompt problems. Be honest about whether you've actually exhausted the cheap, reversible levers first.)
Sources
Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685, 2021) (https://doi.org/10.48550/arXiv.2106.09685)
Hugging Face PEFT documentation: parameter-efficient fine-tuning (https://huggingface.co/docs/peft/index)
Luo et al., An empirical study of catastrophic forgetting in large language models during continual fine-tuning (IEEE Transactions on Audio, Speech and Language Processing, 2025) (https://doi.org/10.1109/TASLPRO.2025.3606231)
Optional J. Indie Hacking with AI
Solo founders, shipping fast, distribution, bootstrapping.
Build and sell software as a one-person company powered by AI.
By the end of this chapter, you will be able to
Explain how AI lowers the team size needed to ship a product solo
Identify a repeatable distribution channel suited to a specific product and niche
Compare bootstrapping and fundraising as growth strategies for a solo founder
J.1 Solo founders shipping quickly
Indie hacking (building and selling software as a one-person or tiny company) has been transformed by exactly the skills this book teaches [1]. AI assistance lowers the team size needed to ship a real product: a single founder can now credibly cover product, frontend, backend, and basic design by leaning on AI for the first draft in each area, then applying judgment to make it actually good, so roles that used to require hiring are now things one person can span.
Speed is the resulting superpower, and for a solo founder it's the main competitive edge. A team of one has no meetings and no coordination cost. Decisions happen as fast as one person can make them. Paired with AI doing the heavy lifting on implementation, that means shipping and iterating weekly, and a founder who learns from real users every week can out-maneuver a larger, slower competitor who learns every quarter.
The catch is that the same leverage is available to everyone, so building fast is necessary but not sufficient. When anyone can spin up a working product in a weekend, execution speed alone is no longer a moat, which throws the weight onto the parts AI doesn't do for you: choosing the right problem (Chapter 12), reaching customers, and finding defensibility (the Feature Graveyard lesson in 12.6). The next two sections are about exactly those non-coding skills that now decide who wins.
Build โ Launch โ Market โ Earn โ (repeat)
The solo founder's loop. With AI writing the code, the bottleneck moves to launch and distribution.
A common pricing ladder. Each tier serves a different customer and a different stage of the business.
J.2 Distribution
Here's the lesson that humbles most technical founders: a great product with no distribution plan gets zero users. Building is now the easy part, and 'if I build it, they will come' is the expensive myth that kills more indie projects than bad code ever has. Getting the product in front of people is a separate discipline, and usually the harder one.
Indie hackers win distribution by going narrow and repeatable, not broad and expensive. A solo founder can't outspend anyone on ads, so the playbook is the unscalable one [3]: build in public (sharing progress and metrics to attract an audience that becomes your first users), embed in the niche community where people with the problem already gather, and produce content tied directly to the problem you solve. The aim is one or two channels you can run yourself, every week, with predictable results.
'Repeatable' is the operative word, because a one-time spike (a post that happens to go viral) is luck you can't reproduce rather than a distribution strategy. A real channel is one where doing the same thing next week yields a similar result, so growth compounds instead of relying on lightning striking twice. Finding and committing to such a channel, for a product targeted at a reachable niche (Chapter 12's reachable-users criterion), is what turns a built product into a growing one.
J.3 Bootstrapping
Bootstrapping means funding growth from your own revenue instead of outside investment, which makes it the natural fit for indie hacking [2]. The trade is straightforward: you keep full ownership and control and answer to no one, but you cap how fast you can spend to grow, since every dollar you reinvest first has to be earned. For many founders, that control is the entire point.
It works best for focused, well-scoped products that can reach profitability with a modest number of paying customers, which is exactly what the MVP discipline of Chapter 12 produces, so you don't need a million users. You need enough customers paying enough to cover costs and pay you, and the key metric becomes MRR (monthly recurring revenue), the predictable income that tells you the business is actually alive rather than just busy.
The AI era makes bootstrapping more viable than it's been in years, which ties this whole chapter together. Because AI slashes both the team size and the cost of building and running software, the revenue bar for profitability drops with it. A one-person company with low costs can sustain itself on numbers that would have looked like failure a decade ago. That's the optimistic core of indie hacking now: the leverage that lets you build alone is the same leverage that lets you stay independent.
Summary
AI lets one person credibly span a whole product. The hard part shifts from building to getting users. We covered going narrow and repeatable on distribution, bootstrapping from revenue to keep ownership, and watching monthly recurring revenue as the health metric. A one-time viral spike is luck, not a strategy. The operative word: repeatable.
Key terms
Indie hacker. A solo or very small founder building and selling software independently, often profitably and without outside funding.
Solo founder. A single person who now credibly covers product, frontend, backend, and basic design by leaning on AI for a first draft in each area, then applying judgment to make it good, spanning roles that used to require hiring a team.
Ship velocity. The pace at which a solo founder can build, launch, and iterate. With no meetings or coordination cost, decisions happen as fast as one person can make them, but since the same speed is available to everyone, velocity alone stops being a moat once anyone can spin up a product in a weekend.
Distribution. The way you get a product in front of people. Without a plan for it, even a great product gets zero users.
Build in public. Sharing your progress, metrics, and process openly to attract an audience and early users.
Niche. A specific, reachable group of people with a shared problem, small enough that a solo founder can find a repeatable channel to reach it directly rather than competing broadly on ad spend.
Bootstrapping. Growing on revenue rather than investment, keeping control but capping spend.
MRR (Monthly Recurring Revenue). Predictable subscription revenue per month, a core health metric for indie SaaS.
Check your understanding
Q1. What has AI assistance specifically lowered for solo founders?
The team size needed for product, frontend, backend, design
The need to have any kind of distribution plan at all whatsoever
The need to ever go out and find any paying customers at all
Nothing meaningful has actually changed for solo founders lately
Answer: A. AI-assisted generation across multiple disciplines lets one person credibly cover ground that used to require a small team.
Q2. Why does this chapter say build speed alone is 'no longer a moat' for a solo founder?
Because building software is now impossible without a full team
Because everyone has the same leverage, so choice and distribution decide
Because quickly built products reliably carry more bugs
Because moats no longer matter for any modern business
Answer: B. When anyone can spin up a working product in a weekend, execution speed stops differentiating founders. What decides the winner is choosing the right problem, reaching customers, and finding real defensibility.
Q3. What makes a distribution channel 'repeatable' rather than lucky, per this chapter?
It generated one enormous one-time spike in traffic
It depends on having a large advertising budget
The same effort next week yields a similar result each time
It came recommended to you by one of your investors
Answer: C. A one-time viral spike is luck you can't reproduce. A real channel produces a similar result if you run the same playbook again, which is what lets growth compound.
Practice
Exercise 1. Pick a real product idea and name one specific, repeatable distribution channel you could realistically run solo for it (not 'go viral'). Explain why that channel fits the audience. (Hint: A channel is repeatable if you could do it again next week with roughly the same effort and expect a similar result.)
Exercise 2. For a small SaaS idea, estimate the number of paying customers at a specific price you'd need to reach a sustainable MRR as a solo bootstrapped founder, assuming AI keeps your running costs low. Then state one reason you'd choose to bootstrap this instead of raising money. (Hint: Work backward from a modest, believable monthly income goal, not a huge one. Bootstrapping math works because the AI era dropped the customer count needed for a one-person company to feel alive.)
Sources
Indie Hackers: the build-in-public community and its MRR-reporting culture (https://www.indiehackers.com/)
Jarvis, Company of One: why staying small can be the point (2019) (https://openlibrary.org/works/OL19799434W)
Graham, Do Things That Don't Scale (July 2013) (https://paulgraham.com/ds.html)
How AI reshapes jobs, productivity, and the human-AI working relationship.
By the end of this chapter, you will be able to
Distinguish tasks likely to be automated from tasks likely to be amplified
Hold both displacement and new-opportunity effects of AI on the labor market honestly
Explain the centaur (human-AI collaboration) model of knowledge work
K.1 What gets automated vs. amplified
The single most useful distinction for thinking about AI and work is automation versus amplification [4]. A task is automated when a person stops doing it entirely because a system does it end to end. It's amplified when a person still does it but faster or at higher quality, with AI handling the rote portion. Most of the public conversation fixates on automation, and for the first years of the technology, usage data put amplification well ahead. In mid-2025 Anthropic's own usage index showed automation-style use briefly pulling ahead of amplification among consumer users, and by late 2025 amplification was back in front at roughly 52 percent against 45 percent, while business API traffic stayed the more automated of the two throughout. The distinction still matters more than the ratio, and the ratio will keep moving.
Which one a given task falls into is roughly predictable. Tasks with a single, clearly checkable correct answer drift toward automation, because a system can be trusted to just do them. Tasks requiring judgment about ambiguous trade-offs drift toward amplification, because the human still has to make the call even when AI does the legwork. Most jobs are a bundle of both kinds of tasks, which is why whole roles rarely vanish overnight even as their day-to-day changes a lot.
This distinction tells you which skills are worth building, and the answer is counterintuitive. As execution speed stops being the bottleneck, raw production skill matters less and judgment matters more. For an amplified task, being able to evaluate, direct, and correct the AI's output becomes the high-value skill, exactly the verification ability this whole book trains. The work shifts from doing the thing to deciding whether the thing was done right.
What moves to machines and what stays human. The line keeps moving right, but the right end stays occupied.
The honest split. Routine output gets automated; taste, expertise, and direction get amplified.
K.2 Labor market impacts, honestly
Honest discussion of AI's labor effects requires holding two true things at once, which is hard because each side of the debate prefers only one of them. Controlled studies have found real productivity gains from generative AI, on the order of forty percent less time on writing tasks [2] and roughly fifteen percent more issues resolved per hour in customer support, concentrated among less experienced workers [1]. A randomized trial with experienced developers found the opposite; Chapter 9 uses that study to explain overreliance [5]. It's true that some roles are genuinely shrinking where AI substitutes well for the work, and pretending otherwise is dishonest. It's also true that AI is creating real new opportunity by making previously uneconomical products and services viable, and the doomers ignore that half.
The uncomfortable part is that these two effects don't cancel out neatly, because they often land on different people. The displacement and the new opportunity rarely accrue to the same individuals, the same regions, or the same skill sets, so 'it all balances out in aggregate' can be cold comfort to someone whose specific role shrank. An honest account acknowledges the transition has real costs even if the long-run picture includes real gains.
For you, the practical response is to aim at where the work is going, not where it has been. That means leaning into the judgment-and-direction skills that amplification rewards, staying willing to reskill as the boundary between automated and amplified keeps moving, and treating adaptability itself as the durable asset, since the specific tasks that are safe will shift, while the ability to keep learning where human value sits doesn't.
K.3 Human-AI collaboration as the default mode
Strip away the speculation, and the most likely near future of knowledge work is neither humans replaced nor humans working alone. It's collaboration, a human directing and checking an AI that does much of the production. Economists describe this as technology displacing labor in some tasks while reinstating it in others [3]. Sometimes this is called the centaur model, after chess teams of human plus engine that outperformed either alone [6], and it's the shape most jobs are quietly converging toward.
You already know this pattern intimately, because it's the vibecoding loop from Chapter 1 generalized beyond code. Intent, generation, verification, refinement (a human owning the goal and the judgment while AI owns the first draft) describes a lawyer reviewing AI-drafted contracts, an analyst checking AI-built models, a marketer editing AI copy, just as well as it describes building software, so this book was, in a sense, training for the general case all along.
That makes the durable career skill clear: be a good director. That means clear specification (saying precisely what you want), sharp judgment about output (knowing good from plausible-but-wrong), and the wisdom to know when to take the wheel back and do something yourself. These are the abilities that stay valuable as the tools underneath them keep changing. The specific AI you'll direct in five years doesn't exist yet, but the skill of directing one well is the thing this whole book has been quietly building, and the thing worth carrying forward.
Summary
The useful question isn't whether AI takes jobs but which tasks it automates versus amplifies. We covered that amplification is more common than the headlines suggest, that displacement and new opportunity are both real and land on different people, and that knowledge work is converging on the centaur model: a human directing and checking an AI. Being a good director, with clear specification plus sharp judgment, is the skill that outlasts the tools.
Key terms
Automation. A task that stops being done by a person at all, because a system does it end to end.
Augmentation. A task a person still does, but faster or better with AI handling the rote part.
Productivity. Output per unit of human effort. AI raises it primarily through amplification (faster, higher-quality work with a human still directing), which leads consumer usage in the latest figures, while business API traffic skews toward automation.
Labor market. The honest accounting of AI's effect on jobs: some roles genuinely shrink where AI substitutes well, while AI also creates real new opportunity by making previously uneconomical products viable, with displacement and new opportunity rarely landing on the same people.
Displacement. The loss of specific roles where AI fully substitutes for the work, part of an honest accounting alongside new opportunity.
Reskilling. Workers learning new skills to move into roles that AI created or left intact.
Collaboration. The near-future default of knowledge work: a human directing and checking an AI that does much of the production, generalizing the intent-generation-verification-refinement loop from Chapter 1 beyond code.
Centaur model. A human-AI pairing where the human directs and judges while the AI executes, the durable mode of most knowledge work.
Check your understanding
Q1. What's the key difference between a task that's automated versus one that's amplified?
There's no structural difference, only a difference in branding
Amplified tasks are reliably of lower quality overall
Automation applies only to physical and manual labor
Automated tasks leave the person entirely; amplified ones keep them
Answer: D. Automation removes the human from the task entirely. Amplification keeps a human directing it while AI handles the rote portion.
Q2. Why does this chapter insist on holding both displacement and new opportunity as true at the same time?
Because both are real and land on different people, so aggregates mislead
Because they cancel out exactly, so neither one really matters
Because displacement is the only real effect and opportunity a myth
Because new opportunity is the only real effect and displacement is exaggerated
Answer: A. An honest account acknowledges that the transition has real costs for specific people even though the long-run aggregate picture includes real gains for others.
Q3. What's the 'centaur model' of knowledge work, and what skill does it reward?
A model in which the AI works entirely unsupervised throughout
A human directing and checking an AI, rewarding specification and judgment
A model in which two AIs collaborate with no human input at all
A purely automated pipeline with no human role at all
Answer: B. Named for human-plus-chess-engine teams that outperformed either alone, the centaur model rewards being a good director: specifying clearly, judging output sharply, and knowing when to take the wheel back.
Practice
Exercise 1. Pick a job you're familiar with (yours, a family member's, or one you've researched). List two tasks within it that are more likely to be automated versus amplified, with your reasoning. (Hint: Tasks with a single, clearly checkable correct answer tend toward automation. Tasks requiring judgment about ambiguous trade-offs tend toward amplification.)
Exercise 2. Describe what the 'centaur' version of a job you're familiar with looks like three years from now: what does the human own, and what does the AI own? Then name the one directing skill (clear specification, sharp judgment about output, or knowing when to take the wheel back) that role most needs to build. (Hint: The vibecoding loop from Chapter 1, intent, generation, verification, refinement, is the template. Map that job's version of each step onto human versus AI.)
Sources
Brynjolfsson, Li and Raymond, Generative AI at Work (Quarterly Journal of Economics, 2025; NBER WP 31161, 2023) (https://doi.org/10.1093/qje/qjae044)
Noy and Zhang, Experimental evidence on the productivity effects of generative AI (Science, 2023) (https://doi.org/10.1126/science.adh2586)
Acemoglu and Restrepo, Automation and New Tasks: How Technology Displaces and Reinstates Labor (JEP, 2019) (https://doi.org/10.1257/jep.33.2.3)
Anthropic Economic Index: the amplification and automation split across Claude.ai and business API traffic (January 2026 report, covering November 2025 data) (https://www.anthropic.com/research/anthropic-economic-index-january-2026-report)
METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (July 2025) (https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)
Kasparov, The Chess Master and the Computer (New York Review of Books, 2010) (https://www.nybooks.com/articles/2010/02/11/the-chess-master-and-the-computer/)
Optional L. History of AI Hype Cycles
Expert systems, AI winters, deep learning, generative AI.
Past hype cycles teach humility and pattern-recognition for the present one.
By the end of this chapter, you will be able to
Summarize the expert-systems AI winter and what caused it
Explain how scaling laws drove the deep-learning and generative-AI wave
Apply historical hype-cycle skepticism to claims about AI's near-term future
L.1 Expert systems and AI winters
AI didn't begin in 2022. The field named itself at a 1956 Dartmouth workshop [1], and knowing its earlier cycles is a vaccine against believing the current one is unprecedented. In the 1980s, rule-based expert systems (programs that encoded human expertise as hand-written if-then rules, a paradigm called symbolic AI) generated enormous investment and confident promises that human-level reasoning was just around the corner. For a while they genuinely worked on narrow problems, which made the promises feel credible.
Then reality arrived. Expert systems turned out to be brittle: they couldn't generalize beyond the exact domains their rules were written for, and maintaining the rules grew unmanageable as problems got complex. A damning 1973 UK government review had already helped trigger an earlier funding collapse [2]. The gap between the promise and the delivery became undeniable, funding and interest collapsed, and the field entered an 'AI winter,' a prolonged bust that followed the boom.
The pattern is what matters, because it has repeated. A real but narrow capability gets oversold as something close to general intelligence, a shape formalized as the hype cycle [6]. The overselling outruns the delivery, and then disillusionment and a funding collapse follow. This wasn't a one-time error. It's a rhythm the field has moved through more than once, and recognizing the rhythm is the whole reason this optional chapter exists.
AI has boomed and busted before. Knowing the pattern makes the current moment easier to read.
Trigger โ Peak hype โ Trough โ Plateau
Every breakthrough technology rides this curve. Useful work happens on the plateau, not the peak.
L.2 Deep learning and generative AI
The thaw came from a different approach. Rather than hand-coding rules, deep learning uses many-layered neural networks that learn patterns from data, and its 2010s resurgence was driven less by a single brilliant new idea than by two quantitative changes [3]: far more data and far more compute became available.
The transformer architecture arrived in 2017 [4], and what followed was the era of scaling laws, the empirical finding that model capability improves predictably as you add data, parameters, and compute [5], so pouring in more of each reliably produced a better model. That observation, more than any single conceptual breakthrough, drove the path from early deep learning to the large language models and generative AI this entire book is built on, and the current wave is largely that scaling story playing out.
Here's the crucial nuance for keeping perspective: each past wave, including the ones that ended in winters, left behind real and lasting capability, and deep learning didn't evaporate but became infrastructure, so the honest read of history isn't that AI booms are fake. They consistently deliver genuine, durable progress while the hype around them simultaneously overshoots what that progress will do on a near-term timeline, and holding both of those at once is the skill.
L.3 Lessons from the pattern
So what does this history actually teach, given that the current wave is plainly delivering real value? The lesson is calibration, not cynicism. 'AI progress is fake' is the wrong lesson, contradicted by every cycle leaving lasting capability behind, very much including this one. The right lesson is subtler and more useful: the gap between current capability and the most exciting claims about it tends to be wider during a hype peak than it feels from inside the peak.
This is the 'looks right' syndrome from Chapter 9, scaled up from a single output to an entire field. Just as a fluent wrong answer disarms your scrutiny, a field-wide narrative of inevitable imminent breakthroughs disarms your skepticism precisely when valuations, promises, and timelines are at their most stretched. The same calibration discipline applies: separate what's verifiably true today from what's being confidently asserted about next year.
Practically, that makes you both a better builder and a steadier participant. You can be genuinely excited about what AI does now (and this book is evidence it does a great deal) while staying grounded about specific predictions of what it'll do soon, which is exactly the calibrated stance this book cultivates, because history doesn't tell you this wave will end in a winter. It tells you to hold your timeline estimates loosely and your verification habits tightly, which is good advice in any weather.
Summary
AI didn't begin in 2022, and past cycles teach humility. We traced expert systems and the AI winter, the deep-learning thaw driven by more data and compute, and the scaling-law era behind today's models. Every wave left durable capability behind even as its hype overshot the near term. That's the calibrated stance to carry into the present one.
Key terms
Expert systems. 1980s rule-based AI that encoded human expertise as hand-written rules. Their failure to generalize triggered an AI winter.
Symbolic AI. An earlier paradigm representing knowledge as explicit symbols and logic rules, contrasted with learned neural approaches.
AI winter. A period of collapsed funding and interest after AI overpromised and underdelivered.
Deep learning. Machine learning using many-layered neural networks, whose 2010s resurgence (more data and compute) led to today's models.
Neural network. A many-layered learning system that finds patterns in data instead of following hand-written rules, the foundation deep learning is built on in place of symbolic AI's explicit logic rules.
Generative AI. The current wave of AI, built on transformers and scaling laws, that this entire course is built on: large language models and related systems that generate text, code, images, and more from a prompt.
Transformer. The neural network architecture introduced in 2017 that underlies modern large language models.
Scaling laws. Empirical findings that model capability improves predictably with more data, parameters, and compute, a driver of the current wave.
Hype cycle. The recurring pattern where excitement about a technology overshoots its near-term capability before settling.
Check your understanding
Q1. What actually happened after each past AI hype cycle, according to this chapter?
The field collapsed entirely and never recovered afterwards
Nothing of any lasting value was produced by the earlier waves
Durable capability remained even as the loudest claims didn't pan out
Hype cycles are a new phenomenon unique to generative AI
Answer: C. Expert systems and deep learning both left real, lasting capability behind even though the loudest claims at each peak overshot what was actually delivered on that timeline.
Q2. What actually caused the 1980s expert-systems AI winter, according to this chapter?
A sudden loss of interest in AI research, with no prior cause
Computers became far too expensive to run any AI software on
Governments banned academic AI research more or less entirely
They proved brittle beyond their hand-written rules
Answer: D. The gap between the promise (human-level reasoning just around the corner) and the delivery (brittle, narrow, hard-to-maintain rule systems) became undeniable, and funding collapsed as a result.
Q3. What drove the deep-learning resurgence in the 2010s, per this chapter?
Two shifts: far more data and compute, then the transformer
One brilliant new algorithm, invented more or less overnight
Government mandates requiring firms to adopt neural networks
The complete abandonment of all rule-based approaches
Answer: A. The chapter is explicit that the resurgence was driven less by one conceptual breakthrough than by more data and compute, with the transformer later giving that approach a form that scaled predictably.
Practice
Exercise 1. Pick one bold claim currently being made about AI's near-term future. Compare it to a claim made during a past hype cycle (expert systems or early deep learning) and note what's structurally similar. (Hint: Look specifically at the gap between 'what the technology can verifiably do today' and 'what the claim says it will imminently do.' That's the gap that mattered in past cycles too.)
Exercise 2. Explain in your own words why expert systems failed to generalize, and identify one thing the deep-learning wave did differently that avoided that specific failure mode. Then name one piece of durable capability that survived the expert-systems era despite the winter. (Hint: The brittleness came from hand-written rules that couldn't cover cases nobody anticipated. Deep learning's shift to learning patterns from data, instead of encoding them by hand, is the structural difference worth naming.)
Sources
McCarthy, Minsky, Rochester and Shannon, A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence (1955) (http://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html)
Lighthill, Artificial Intelligence: A General Survey (Science Research Council, 1973) (http://www.chilton-computing.org.uk/inf/literature/reports/lighthill_report/p001.htm)
Krizhevsky, Sutskever and Hinton, ImageNet Classification with Deep Convolutional Neural Networks (NeurIPS 2012) (https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html)
Vaswani et al., Attention Is All You Need: the transformer architecture (NeurIPS 2017) (https://doi.org/10.48550/arXiv.1706.03762)
Kaplan et al., Scaling Laws for Neural Language Models (arXiv, 2020) (https://doi.org/10.48550/arXiv.2001.08361)
Fenn and Raskino, Mastering the Hype Cycle (Harvard Business Press, 2008) (https://openlibrary.org/works/OL11983657W)
Optional M. The Toolbox as a Working Reference
Install commands, the model list, effort settings, and a tool comparison, in one place to look up.
Reference chapter. Look things up here as you need them; it isn't meant to be read straight through.
This is a lookup chapter, not a narrative one. Everything here is the kind of detail you check while setting up or deciding which tool and model to point at a task: install commands, the current models with prices, what the effort setting does, and how the major tools compare. Facts about a fast-moving field go stale, so each one is dated. Treat those dates as expiration warnings and confirm against the linked sources before you rely on a number.
By the end of this chapter, you will be able to
Install Claude Code and an AI-native IDE on your own platform
Pick a model by matching capability and cost to the task in front of you
Set an effort level appropriate to the work, and recognize when more effort stops helping
Compare the major AI coding tools and their student offers at a glance
M.1 Installing the tools
Claude Code is a terminal-native agent. As of mid-2026, install it with one of the following. On macOS, Linux, or WSL, run curl -fsSL https://claude.ai/install.sh | bash [1]. In Windows PowerShell, run irm https://claude.ai/install.ps1 | iex. In Windows CMD, run curl -fsSL https://claude.ai/install.cmd -o install.cmd && install.cmd && del install.cmd. Package managers also work: brew install --cask claude-code on macOS, or winget install Anthropic.ClaudeCode on Windows. After it installs, run claude inside any project folder and log in on first use.
Claude Code needs a paid account to run: a Claude Pro, Max, Team, or Enterprise subscription, or an Anthropic Console account billed by usage. The same engine also ships as a desktop app, a VS Code and JetBrains extension, and a browser version at claude.ai/code, all sharing your CLAUDE.md files and settings so work moves between them.
Cursor is a full AI-native editor you download and install like any other app from cursor.com [4], then open your project inside it. The other tools in this space install the same way from their own sites.
Installing the Claude Code extension from the VS Code marketplace: search 'claude code' in the Extensions panel and install the one published by Anthropic.
M.2 The current models
Model names and prices move quickly, so read the table below as a snapshot dated 2026-08-07 [3]. Input tokens are what you send the model, output tokens are what it generates, and output is consistently the more expensive half. When a tool asks which model to use, pass the model id, not the marketing name.
Two practical notes sit on top of that table. First, the most capable model isn't always the one you can select. Fable 5 became generally available on June 9, 2026 and is the strongest widely released model, while Mythos 5 shares its specs and pricing but stays invitation-only for approved customers doing defensive cybersecurity work. Opus 5 is the practical default for serious agentic coding, Sonnet 5 is the cheaper choice for lighter edits and high-volume work, and Haiku 4.5 is cheaper still for the simplest requests. Second, match the model to the task: reaching for the largest model to rename a variable wastes money, and reaching for the smallest to design a schema wastes the afternoon you then spend fixing it.
Model
Model id
Context
Input $/1M
Output $/1M
Claude Fable 5
claude-fable-5
1M
$10
$50
Claude Opus 5
claude-opus-5
1M
$5
$25
Claude Sonnet 5
claude-sonnet-5
1M
$2
$10
Claude Haiku 4.5
claude-haiku-4-5
200K
$1
$5
Claude Code's model picker, showing the current models with Opus 5 selected alongside a link to the availability notice.A multi-provider tool like Cursor exposes models from several labs at once, each with its own context window and effort level. The model id still matters more than the marketing name.
M.3 Match thinking to the task
Beyond which model you pick, several tools let you set how much effort the model spends reasoning before it answers, on a ladder that runs from low through medium, high, and xhigh to max. More effort means the model thinks longer and usually answers better on hard problems, at the cost of time and tokens, with high as a sensible default. High is the default in Claude Code and on the API, and it is the right setting for most work. Step up to xhigh for demanding coding and long-running agentic tasks, where the deeper deliberation earns its higher token spend.
The top of the ladder isn't a free upgrade. The max setting is reserved for genuinely hard problems, and on easy ones it can overthink, talking itself out of a correct simple answer. The skill is matching effort to the task, the same judgment choosing a model asks of you, applied to thinking time instead of model size. Effort is also a separate thing from spawning subagents to divide a job up: it's about how long one model deliberates. In this build, the label 'ultracode' names a Claude Code setting that pairs xhigh effort with automatic workflow orchestration, not a different or smarter model.
Claude Code's mode menu and the Effort selector. 'Ultracode' here is xhigh effort plus automatic workflow orchestration, and 'Bypass permissions' at the bottom is the approvals-off setting Chapter 3 covers.
M.4 The tools at a glance
Four tools cover most of what this book touches, and they differ less in raw capability than in shape, summarized in the table below. The right one is mostly the one that fits how you already work, and trying two is the fastest way to feel the difference.
Student and trial offers are worth checking because they change often. As of mid-2026, Cursor has retired the free year it once gave verified students and now points them to a free tier plus credits handed out at campus and online events [5], Codex has a free plan with limited usage, and Antigravity is free for individual developers. These promotions expire and shift, so confirm the current terms on each tool's own site rather than trusting a number printed in a book.
Parallel sessions, and comparing models on one prompt. Shipped as Windsurf until June 2026
Tool
How it runs several at once
How the parallel work is kept apart
Claude Code
Subagents inside a session, dynamic workflows, and experimental agent teams
Git worktrees, optionally one per agent
Codex
Several tasks delegated at once, running in the cloud while you work on something else
An isolated cloud environment per task, and git worktrees locally
Cursor
An agents window running many at once, /multitask, and /best-of-n across models
Git worktrees locally, a VM and a branch per agent in the cloud
Antigravity
A manager surface for spawning and watching agents, plus dynamic subagents
A worktree selector on each new conversation
Devin Desktop
Parallel sessions, and an arena mode that runs one prompt across models
A git worktree per session
GitHub Copilot
An agents panel, batch issue assignment, and /fleet in the CLI
One ephemeral cloud environment per session
Codex's menu, with a Plan mode toggle and a 'Full access' permission indicator. Each tool wraps the same ideas, plan then act under a permission boundary, in its own interface.
M.5 Where the settings live
When you want to change behavior, a few files and commands cover most cases. Persistent instructions hold standing guidance the tool reads on every request in that repo: conventions, the build command, and things to avoid [2]. Chapter 3.3 covers what each vendor calls this mechanism. The file locations move, so check the tool's current docs instead of trusting a filename you remember. Cursor's flat .cursorrules, for instance, has been superseded by a directory of scoped rule files under .cursor/rules. The old filename no longer appears in Cursor's own documentation. Ignore files (.gitignore and tool-specific ignore lists) keep generated output and third-party library code out of the model's context. Permission settings control which actions run automatically and which pause for your approval, and they're where you would tighten or loosen the boundary discussed in Chapter 3.
One small but common confusion: in Claude Code, /model is a slash command you type inside the session to switch models, not a shell command you run in your terminal. Claude Code can also run shell commands for you as part of a task, but that's a separate capability from its own slash commands like /model, /clear, or /help. When in doubt, the settings reference linked at the end of this chapter is the authoritative place to look, precisely because the defaults printed here will drift.
Typing /model in Claude Code opens model and effort settings, here Opus with extra-high effort. /model is a slash command inside the session, not a shell command you run in the terminal.
M.6 The expiration ledger for claims that go stale
This book dates its volatile facts on purpose, and this table is where they all live. Each row is a claim somewhere in the book that was true when written and will eventually stop being true: a price, a default, an availability restriction, a promotional offer, or a tool capability. Historical facts, like the day GitHub stopped accepting passwords, are dated too but don't drift, so they aren't listed here. If you're reading a printed edition, checking this table against the linked sources tells you in one sitting how much the world has moved since the ink dried.
For instructors and for us, the ledger doubles as the maintenance checklist. Refreshing an edition means walking these rows against their verify-against links and updating what changed, instead of re-reading thirty-one chapters hunting for the word 'currently.'
Claim
Where
As of
Verify against
Anthropic models, ids, context windows, and per-token prices
1.7, 1.8, M.2, Q.1, Q.6
2026-08-12
platform.claude.com models overview and pricing pages
Fable 5 generally available; Mythos 5 invitation-only (Project Glasswing)
Design-system context exposed to agents over MCP, meaning component registries and a design tool's variables and component-to-code mappings
5.3
mid-2026
ui.shadcn.com/docs/mcp and the Figma Dev Mode MCP docs
Generator-side design system registration, teaching a UI generator your real components and tokens
5.3
mid-2026
v0.app/docs/design-systems-2
Design Tokens Format Module first stable version, 2025.10, with theming and multi-brand sets
5.3, 5.4
2025-10-28
designtokens.org and the Design Tokens Community Group announcements
Baseline availability of container queries (February 2023) and light-dark() (May 2024)
5.4
mid-2026
developer.mozilla.org and the web platform Baseline data
Agent-driven browser tool surface, meaning viewport resize, color-scheme emulation, screenshot, and Lighthouse audit
5.5
mid-2026
Chrome DevTools MCP tool reference on GitHub
WebAIM Million figures: 95.9 percent of home pages with detected WCAG failures, 56.1 errors per page, 83.9 percent with low contrast, ARIA attributes up 27 percent
5.6
2026-02
webaim.org/projects/million
The WCAG version referenced by the harmonized European standard under the European accessibility directive
5.6 callout
mid-2026
EN 301 549 and the European Commission's European Accessibility Act pages
90 percent of 200 audited AI-agent-built applications carrying at least one vulnerability, broken access control at 36 percent of findings, and 82.8 percent of those failures in backend code
6.1
2026-06
doi.org/10.48550/arXiv.2606.23130
Exploits running successfully against roughly half of the functionally correct backends across 392 benchmark tasks
6.1
2025
doi.org/10.48550/arXiv.2502.11844 and baxbench.com
Firebase offering a managed Postgres option (SQL Connect, renamed from Data Connect in April 2026) alongside Firestore
6.1
2026-08
firebase.google.com/docs/sql-connect
Schema-diffing migration tools emitting a DROP plus an ADD for a renamed field by default
6.3
mid-2026
prisma.io/docs (customizing migrations), plus your own tool's generated migration SQL
Supabase enabling row-level security by default on tables created in the dashboard but not on tables created in raw SQL
Payment processor roster and merchant-of-record status (Stripe, PayPal/Braintree, Square, Paddle, Lemon Squeezy, Adyen)
7.4
2026-08
Each provider's own pricing and merchant-of-record pages, starting with stripe.com/managed-payments and paddle.com/pricing
AI company gross margins averaging about 45 percent in 2025 with about 53 percent projected for 2026
A.2
2026-07
iconiq.com/growth/reports/state-of-ai-2026
AI pricing model mix: consumption-based rising from 35 to 42 percent of companies and outcome-based from 18 to 23 percent over six months, at an average of 1.7 models per company
A.1
2026-07
iconiq.com/growth/reports/state-of-ai-2026
Fastest-growing AI companies running near 25 percent gross margin, with the tier below them near 60 percent
A.2
2025-08
bvp.com/atlas/the-state-of-ai-2025
Effective cost per output token varying by a factor of 2.5 to 36 on identical hardware depending on utilization
A.2
2026-06
doi.org/10.48550/arXiv.2606.11690
Coding assistants metering premium requests once an included monthly allowance runs out
Student and trial offers: Cursor free tier plus event credits, Codex free plan, Antigravity free for individual developers
M.4
2026-08-12
Each vendor's own pricing and student pages
Vendor names and file locations for persistent instructions, including Cursor's move from .cursorrules to a .cursor/rules directory
M.5
2026-08
cursor.com/docs/context/rules, code.claude.com/docs/en/memory, and docs.github.com (Copilot custom instructions)
What Claude Code ingests: permission-gated web fetch outside a preapproved domain list, browser control through the Chrome extension, and images read from disk
11.6
mid-2026
code.claude.com/docs/en/tools-reference and code.claude.com/docs/en/chrome
Codex CLI defaults to cached web search; live fetching and cloud network are opt-in
11.6
mid-2026
learn.chatgpt.com/codex/web-search
Codex's built-in browser exists in ChatGPT on the web and in the desktop app, but not in the CLI or IDE extension
11.6
mid-2026
learn.chatgpt.com/codex/browser
Claude and OpenAI APIs accept no native video input; Gemini reads mp4 with both streams
11.6
mid-2026
Each provider's file-input and video docs
GitHub authentication options and push-protection defaults
P.4, P.6
mid-2026
GitHub docs: authentication, push protection
Secret scanning via the GitHub MCP server generally available
P.6
2026-05-05
GitHub changelog
Claude subscription prices and rolling usage-window structure
Q.5
mid-2026
claude.com/pricing and support.claude.com
Prompt-cache multipliers (1.25x/2x write, 0.1x read) and per-model minimums (512 Opus 5, 1,024 Sonnet 5, 4,096 Haiku 4.5)
Q.3
2026-08-12
platform.claude.com prompt-caching docs
Batch API flat 50 percent discount
Q.4
mid-2026
platform.claude.com batch-processing docs
Cache reads not counted against input-token rate limits
Q.3
mid-2026
platform.claude.com rate-limits docs
Amplification ahead of automation on Claude.ai at about 52 against 45 percent, with business API traffic more automated
K.1
2026-08
anthropic.com Economic Index reports
OpenClaw naming (ex-Clawdbot/Moltbot), channel list, and onboard flow
8.6
mid-2026
docs.openclaw.ai and the project GitHub
OpenClaw Foundation launched July 8, 2026 as the project's steward
OSWorld past its roughly 72 percent human baseline, with top runs above 85 percent and OSWorld 2.0 best at about 21 percent
8.7
2026-08
osworld-v1.xlang.ai and osworld-v2.xlang.ai leaderboards
Provider defaults on training and retention for API inputs, including the 30-day deletion window
14.1
2026-08
privacy.claude.com data-use and data-retention articles
Thomson Reuters v. Ross fair-use ruling on appeal, argued in the Third Circuit June 2026 with a decision pending
14.3
2026-08
Third Circuit docket No. 25-2153
Data-center share of global electricity and the AI-driven growth rate
14.5
2026-08
iea.org/reports/key-questions-on-energy-and-ai
A row changed? Tell usIf you check a row and the world has moved, the fastest way to get the book fixed is an issue on the textbook's GitHub repository naming the row and what you found. The web edition updates continuously; printed editions absorb the ledger once per printing.
Summary
This reference gathers the practical details in one place: install commands, the current models with prices, what the effort setting does, a comparison of the major tools, and where settings live. Everything here is dated because it goes stale fast, so confirm against the linked sources before relying on a number. The recurring judgment: match the model and the effort to the task, not always the biggest. The chapter closes with the expiration ledger, a single table tracking every dated claim in the book and where to verify it.
Key terms
Install command. The platform-specific command that sets up a tool like Claude Code, such as curl -fsSL https://claude.ai/install.sh | bash on macOS/Linux/WSL, or a package manager like brew install --cask claude-code.
Model id. The exact string you pass to select a model, like claude-opus-5, as opposed to its marketing name.
Context window. The maximum amount of text, measured in tokens, that a model can consider at once, including your prompt and its own output.
Input vs. output tokens. Input tokens are what you send the model. Output tokens are what it generates, billed at a higher rate.
Effort level. A setting (low, medium, high, xhigh, max) for how long a model deliberates before answering. More helps hard problems and can overthink easy ones.
xhigh. The effort setting one notch below max on the ladder, and Claude Code's default for coding because code rewards the extra deliberation. In this build, 'ultracode' names a Claude Code setting that pairs xhigh effort with automatic workflow orchestration, not a different or smarter model.
Rules file. A per-repo file of standing instructions that an AI tool applies to every request in that project. The name differs by vendor (Claude Code memory, Cursor project rules, Copilot custom instructions), but the mechanism is the same.
Ignore file. A file (.gitignore and tool-specific ignore lists) that keeps generated output and third-party library code out of the model's context.
Slash command. A command typed inside an AI tool's session, like /model or /clear, that controls the tool itself, distinct from a shell command run in your terminal.
Check your understanding
Q1. As of mid-2026, which model is the practical default for serious coding, and why not the single most capable one?
Fable 5, because it's the most capable model that's available
Opus 5, because Fable 5 costs twice as much per token for the same job
Haiku 4.5, because it's by far the cheapest option going
Whichever model has the largest context window, whatever the cost
Answer: B. Fable 5 is the most capable widely released model and became generally available in June 2026, but it costs $10 and $50 per million input and output tokens against Opus 5 at $5 and $25. Opus 5 is the default for serious coding because it balances that depth against price.
Q2. What does raising the effort level from high to max actually do?
Switches you over to a larger and smarter model
Spawns several subagents to divide up the work
Makes the same model think longer, and overthink easy problems
Always yields a better answer at no additional cost
Answer: C. Effort controls how long one model thinks before answering. It's separate from model size and from spawning subagents, and at the top of the ladder it can talk itself out of a correct simple answer.
Practice
Exercise 1. Install Claude Code (or an AI-native IDE) on your own machine using the command for your platform, then run it on an empty folder and confirm it starts and asks you to log in. (Hint: Match the command to your shell. The CMD and PowerShell commands look different, and the tool's own error messages tell you which shell you're actually in.)
Exercise 2. Take one real task you did this week and decide, with reasons, which model and effort level you would point at it. Then pick a second task that should use a different setting, and explain the difference. (Hint: The interesting pairs are a cheap simple task next to an expensive hard one. Name what about each task drives the choice.)
Sources
Claude Code install and setup: current commands and package managers (https://code.claude.com/docs/en/overview)
Claude Code settings reference: rules, ignore, and permission configuration (https://code.claude.com/docs/en/settings)
Claude models overview: current lineup, context windows, and pricing (https://platform.claude.com/docs/en/about-claude/models/overview)
Cursor: download and product overview (https://cursor.com)
Cursor for students: the free tier and campus event promotions (https://cursor.com/students)
Model Context Protocol (MCP), the open standard for tool and data connections (https://modelcontextprotocol.io)
Optional N. Start Here If You've Never Programmed
The few ideas to hold in your head before Chapter 1, in plain language.
This book asks you to read and judge code, not to write it from memory. A complete beginner can follow it with a little orientation. This short on-ramp gives you that orientation: what code is, what 'running' it means, the handful of words that recur, and how to use an AI tool without feeling lost. Read it once, then start Chapter 1.
By the end of this chapter, you will be able to
Explain in plain terms what code is and what running it does
Recognize the recurring vocabulary (function, file, terminal, error) when you meet it
Set up one AI tool and follow the course loop with no prior programming experience
N.1 What code actually is
Code is a set of precise instructions a computer follows exactly, with no common sense to fill in what you left out. A program is many of those instructions stacked together to do something useful, like add up a bill or show a page. When people say a program has a bug, they mean there's a gap between what the author meant and the literal instructions the computer actually followed.
The good news for this book is that recognizing whether a result is right is a different and easier skill than producing it from a blank page, and you already make 'is this correct' judgments all the time. If a tip calculator says a 20 percent tip on 50 dollars is 40 dollars, you know something is wrong without writing any code. That instinct, applied a little more carefully, is most of what verification is.
So you don't need to memorize syntax, the exact punctuation and keywords a language uses. The AI handles most of that. What you need is to understand what the program is supposed to do, read the result, and notice when it doesn't match, which is the habit the rest of this book builds on purpose.
Idea โ Prompt โ Run โ Fix โ Share
Your first vibecoding project, end to end. The loop is short on purpose: ship something tiny.
Four habits that keep a beginner on the right side of the toy-versus-product line from day one.
N.2 The words you'll keep hearing
A few terms recur constantly, and knowing them in plain language is enough to follow along. A file is a single document of code or text. A function is a named, reusable chunk of instructions that does one job, like 'calculate the tip.' A variable is a labeled box that holds a value, like the bill amount. Running code means telling the computer to actually carry out the instructions and show you the result.
Two more you'll meet early. The terminal, or command line, is a text window where you type commands to run things instead of clicking buttons. An error message, sometimes shown as a stack trace, is the computer telling you it couldn't do what you asked and roughly where it got stuck. Errors are normal and expected, not a sign you did something wrong, and pasting one back to the AI is a standard way to fix it.
N.3 How to follow along without drowning
Set up one tool and only one to start, an AI-native IDE like Cursor or Claude Code from the reference in Optional M, so you aren't also learning to juggle tools. Begin with the smallest possible thing that runs, then change one small piece at a time, which is exactly the loop Chapter 2 describes, where momentum comes from many tiny working steps rather than one big leap.
Build two habits from day one. First, read every change the AI proposes before you accept it, even if you don't follow every detail, and ask the model to explain anything that's unclear, since it's a patient tutor that never tires of basic questions. Second, when something breaks, read the error, paste it back, and try again. The thing you're working toward is judgment, deciding whether the result is right, and that's a skill you can start building today with zero prior experience.
Summary
A short on-ramp for readers with no programming background. It explains in plain language what code is and what running it means, defines the handful of words that recur (function, variable, file, terminal, error), and gives two habits: reading every change and treating errors as normal. These habits let a beginner follow the course's judgment-first loop.
Key terms
Code. Precise instructions a computer follows exactly, with no common sense to fill in what you leave out.
Program. Many instructions stacked together to do something useful, like calculate a bill or show a page.
Bug. A gap between what the author meant and what they actually told the computer to do.
Function. A named, reusable chunk of instructions that does one job.
Variable. A labeled box that holds a value, like a bill amount.
Terminal (command line). A text window where you type commands to run things instead of clicking buttons.
Error / stack trace. The computer telling you it couldn't do what you asked and roughly where it got stuck. Normal and expected.
Running code. Telling the computer to actually carry out the instructions and show you the result.
Check your understanding
Q1. In this book, what's the main skill a beginner is building?
Memorizing the exact syntax of a whole programming language
Typing out working code as fast as you possibly can
Avoiding mistakes so that no errors ever appear at all
Judging whether a result matches the intent, and correcting it
Answer: D. The course trains judgment and verification, not syntax recall. Recognizing whether output is right is the durable skill, and the AI handles most of the syntax.
Q2. According to this on-ramp, what does a 'bug' actually mean?
A gap between what the author meant and what they actually wrote
A random malfunction with no identifiable underlying cause
A problem that only ever happens in AI-generated code
An error that can never actually be fixed properly
Answer: A. Code is precise instructions with no common sense filled in. A bug is the mismatch between the programmer's intent and the literal instructions given, and the computer does exactly what it was told.
Q3. What does this chapter say about encountering an error message as a complete beginner?
It means something is fundamentally wrong and you should start over
Errors are normal, and pasting one back to the AI is how you fix it
It means the AI tool itself is broken and needs reinstalling
Beginners shouldn't run code until they fully understand it
Answer: B. An error is just the computer reporting it couldn't do what was asked and roughly where it got stuck. Reading it and pasting it back to the AI is the expected next step, not a sign of failure.
Practice
Exercise 1. Open an AI chat assistant and ask it to explain, in plain language, what a 'function' and a 'variable' are, using a tip calculator as the example. Then write one sentence each in your own words. (Hint: If the explanation uses a word you don't know, ask it to explain that word too. Keep going until both sentences feel obvious to you.)
Exercise 2. Ask an AI tool to build the smallest possible working thing (for example, a page that says hello and shows today's date), and deliberately read the change it proposes before accepting it. Write one sentence on what you understood and one question you'd ask the model about a part you didn't. (Hint: You don't need to understand every line. The habit being practiced is reading before accepting, and asking rather than guessing when something is unclear.)
Optional O. Start Here If You're an Experienced Engineer
What actually changes when you already know how to code.
If you can already build software, the risk isn't confusion. It's assuming nothing has changed and coding the way you always have with an autocomplete bolted on. This on-ramp names what genuinely shifts: where your time goes, which instincts still serve you, and which habits now work against you. Then it points you at the chapters worth your time.
By the end of this chapter, you will be able to
Identify which existing skills transfer directly to vibecoding and matter more
Spot the experienced-engineer habits that now cause problems
Choose where to start in the course given what you already know
O.1 What transfers and what shifts
Most of what you already have transfers and is worth more, not less. Your sense of architecture, your debugging instincts, your feel for what 'correct' and 'maintainable' look like, and your knowledge of edge cases are exactly the judgment this workflow runs on. The model is fast at production and weak at judgment, which is the half you're strong at.
What shifts is where your time goes. Typing speed stops being the bottleneck, so the work moves to specification (saying precisely what you want) and review (deciding whether you got it). The reflex worth retiring is 'I could just write this faster myself.' On well-specified, well-trodden tasks the model is good at, that reflex is wrong, though on genuinely novel or subtle ones it's right, and learning to tell those two apart quickly is the new skill.
What changes when you already know how to code. The skills don't disappear; they move up a level.
Where your existing expertise pays off most once the model handles first-draft typing.
O.2 The habits that now hurt
A few veteran habits quietly backfire. The first is checking your own code carefully but accepting a generated change on the strength of its summary, which is backwards, since the generated code is the part you didn't think through. Run it, test it, and read the parts that touch anything that matters. The second is trusting fluent output because it reads like code you would have written, the 'looks right' trap that Chapter 9 makes central, and which bites experts hardest precisely because their pattern-matching is so good.
Two more are about agents. Control instinct makes some experienced engineers under-use agent mode, hand-applying changes an agent would do faster and more consistently. Novelty makes others over-trust it, turning a long unsupervised run loose and accepting whatever it produced. The calibrated middle, steering early and checkpointing often, is Chapter 3. None of these habits reflect a lack of capability. They served you well in a pre-AI workflow and simply need updating.
O.3 Where to jump in
You can move fast through the early material. Skim Chapters 1 and 2 for shared vocabulary and the specific way this book frames prompting and PRDs, then spend your real attention on Chapter 3 (working inside a codebase with agents), Chapter 8 (agent design), and Chapters 9 and 10 (slop and verification), which is the part most likely to catch someone who trusts fluent output. The optional chapters on existing-code work (E) and the toolbox reference (M) will also pay off quickly.
The meta-point is the same one the whole course makes, aimed at you specifically. Your experience is an asset here, but only if you apply it to the model's output as deliberately as you would to a junior engineer's pull request. The moment you start rubber-stamping because the code looks like your own style is the moment the experience stops protecting you.
Summary
A fast-forward for readers who already build software. Your architecture sense, debugging, and edge-case judgment transfer and matter more. Typing speed stops being the bottleneck, and specification and review become the work. It flags the veteran habits that now backfire (skimming generated diffs, trusting fluent output) and points you at Chapters 3, 8, 9, and 10.
Key terms
Specification. Stating precisely what you want built, which becomes the higher-leverage work once typing is no longer the bottleneck.
Review. Deciding whether generated output is actually correct, treated as seriously as reviewing a colleague's pull request.
Looks right trap. Trusting fluent output because it resembles code you would write. It bites experienced engineers hardest (see Chapter 9).
Calibration. Telling apart the tasks where 'I'd be faster by hand' is true from the ones where it's a costly reflex.
Check your understanding
Q1. For an experienced engineer, which habit most often backfires in an AI workflow?
Reading every single line of the generated code carefully
Committing to version control before starting
Skimming a generated diff while scrutinizing their own code
Writing thorough tests for all the tricky edge cases
Answer: C. The generated code is the part you didn't reason through, so it deserves more scrutiny, not less. Skimming it because it looks familiar is the 'looks right' trap.
Q2. For an experienced engineer, where does this chapter say your existing skills (architecture sense, debugging instincts, edge-case judgment) apply most in an AI workflow?
They no longer matter, since the AI now handles all of it
They only matter while writing code entirely by hand yourself
They apply exclusively to specification, and never to code review
They transfer directly, since the model is weak exactly where you're strong
Answer: D. The chapter is explicit that most of what an experienced engineer already has transfers and becomes more valuable, since judgment is the scarce resource in this workflow, not production speed.
Q3. What are the two opposite agent-related habits this chapter says experienced engineers fall into?
Under-using it from control instinct, and over-trusting it from novelty
They refuse to use agent mode at all, without exception
They trust agents completely and immediately, with no oversight
There's one habit only, and it affects everyone identically
Answer: A. Control instinct makes some engineers hand-apply changes an agent would do faster; novelty makes others turn a long unsupervised run loose and accept whatever it produced. The calibrated middle is steering early and checkpointing often.
Practice
Exercise 1. Take a task you'd normally write by hand in a few minutes and instead specify it for an AI tool, then review its output as strictly as a pull request. Note where specifying took longer and where reviewing caught something. (Hint: The interesting data is the tasks where you were tempted to skip the AI. Was the 'I'd be faster myself' instinct right or wrong this time?)
Exercise 2. Using O.3's reading order (skim Chapters 1-2, focus on 3, 8, 9, 10), build a personal two-week reading plan naming the specific chapter or optional appendix you'd read each session and one question you expect it to answer given what you already know. (Hint: Be honest about which of your existing habits (skimming diffs, under- or over-trusting agents) each chapter is meant to correct, and let that drive the order rather than the table of contents.)
Optional P. Git, GitHub, and SSH Keys
Commits as the undo button for agent runs, and the key pair that gets you onto GitHub.
Reference chapter. Look things up here as you need them; it isn't meant to be read straight through.
Most books teach git as hygiene you should adopt because professionals do. This chapter teaches it as infrastructure for working with agents. When a model edits ten files in ninety seconds, the commit you made beforehand is the only thing standing between a bold experiment and a lost afternoon, and the diff it produces is the surface you review the generated code on. SSH keys are the practical gate in front of all of it, because since 2021 there's no way to push to GitHub with a password. Learn the key pair properly once, and you'll never be confused by an authentication failure again.
By the end of this chapter, you will be able to
Explain why commits, branches, and diffs matter more when an agent writes the code than when you do
Run the local git loop confidently: init, status, add, commit, branch, merge, revert
Connect a local repository to GitHub over a remote and read a pull request diff of generated code
Generate an ed25519 SSH key pair, add it to ssh-agent and your GitHub account, and verify the connection
Distinguish the public half of a key pair from the private half and state what each is safe to do
Respond correctly when a secret is committed, starting with rotation rather than history rewriting
Decide which git operations an agent may run unattended and which require your approval
P.1 Why version control matters more when a machine writes the code
Version control was always good practice. Once a model is doing the typing, it becomes the thing that makes the rest of your workflow survivable. A single agent run can touch ten files in ninety seconds, rename a helper you were fond of, quietly delete a test, add a dependency you didn't ask for, and then hand you a summary that sounds entirely reasonable. If you had no commit before that run, the only record of what the project looked like beforehand is your memory of it. If you did have one, the entire run is one command away from never having happened. That asymmetry is the reason this chapter exists.
The habit is small and the payoff is large. Commit, then prompt. A clean working tree before an agent run costs you about four seconds and buys you a guaranteed way back, which in turn buys you permission to be bold. Students who checkpoint reliably try bigger things, because a bad outcome is an inconvenience instead of a disaster. Students who don't checkpoint end up defensive, giving the agent small, timid tasks and hand-applying the rest, which is a slower way to work and a worse way to learn.
The second thing git gives you is a place to read what actually happened. Chapters 9 and 10 ask you to verify generated code instead of trusting it, and git diff is where that verification physically occurs, rather than scrolling through files hoping to notice something. You're looking at a list of exactly the lines that changed, which is a much smaller and much more honest surface than the model's own description of its work, and branches extend the idea to whole sessions. If a two-hour conversation went somewhere useless, you delete the branch, and the main line of your project never knew about it.
The habit this chapter is built around. Four seconds of commit before the prompt is what makes a bold agent run affordable.
Commit before you promptMake a habit of running git status before you start an agent, and committing anything outstanding first. If the working tree is clean when the run begins, then whatever git diff shows afterward is exactly what the agent did, with none of your own half-finished edits mixed in.
If the terminal is new to youEverything in this chapter is typed into a terminal. Optional Chapter N explains what that is and hands you the vocabulary. Read it first if the word is unfamiliar, then come back here.
P.2 The core loop in practice
A git repository is an ordinary folder with a hidden .git directory inside it, created by running git init, or by running git clone on a URL when the project already exists somewhere else [24], and nothing in that folder is tracked until you say so. You stage changes with git add, which selects what goes into the next snapshot, and record them with git commit -m followed by a message [20]. The staging step confuses beginners because it feels redundant, but it's what lets you commit two of the five files you touched, which turns out to matter constantly when an agent has changed more than you meant to keep. Between those two commands sits git status, which is the one you should run more often than you think you need to.
Once you have a history, undoing becomes cheap in several different flavors, and choosing the right one is most of the skill. git restore throws away uncommitted changes to a file and returns it to its last committed state [25]. git reset --hard moves your whole working tree back to a commit, which is the sledgehammer you reach for after a bad agent run, and which permanently destroys anything you hadn't committed. git revert is the polite version: it creates a new commit that undoes an old one, leaving the original in the history. Use revert for anything you've already pushed somewhere other people can see, because rewriting shared history creates a mess for everyone who pulled the old version.
Branches are cheap named pointers into your history. Running git switch -c experiment gives you a place to work where mistakes can't touch your main line, and git merge brings the good version back when it earns its place. For agent work, the pattern that pays off is one branch per session or per feature, so that abandoning a run is a matter of switching back to main and deleting the branch. The table below covers the commands you'll use daily, and everything else in git can wait until you actually need it. The Pro Git book linked at the end of this chapter is where to look when you do.
An undo button is only half of what history gives you. The other half is an archive of every version that ever worked, and with an agent, an archive you can query in plain language. The button that looked better two sessions ago still exists in a commit even though no file on your disk contains it anymore. git log -S finds the commit where a phrase appeared or vanished, git checkout SHA -- FILE resurrects one file from any point in history without moving anything else, and git cherry-pick lifts a single commit's change onto your current branch. You rarely need to remember these incantations, because repository archaeology is a task agents are genuinely good at. Ask for 'the version of the pricing page from before Tuesday's redesign,' and a capable agent will search the history, find the commit, and bring back exactly the piece you missed. This matters more in agent work than it ever did by hand, since a model 'improving' a file routinely overwrites something you liked, and the overwritten version is recoverable only if it was committed, which is one more argument for checkpointing generously.
Command
What it does
When you reach for it
git init
Turns the current folder into a repository
Once, at the start of a new project
git clone URL
Copies an existing remote repository onto your machine
Starting from a repo that already exists on GitHub
git status
Shows what changed and what's staged
Constantly, especially before and after an agent run
git add .
Stages every change in the folder
Right before committing a checkpoint
git commit -m MESSAGE
Records a snapshot with a message
Before a risky prompt, and after a good result
git diff
Shows unstaged line-by-line changes
Reviewing what the model actually wrote
git log --oneline
Lists the commit history compactly
Finding the commit you want to go back to
git switch -c NAME
Creates and moves to a new branch
Starting a session you might want to throw away
git merge NAME
Brings a branch's work into the current one
Keeping a session that worked out
git restore FILE
Discards uncommitted changes to one file
The agent broke one file and you want it back
git reset --hard
Returns everything to the last commit
The agent run was a wash and nothing is worth keeping
git revert SHA
Adds a new commit undoing an old one
Undoing something you already pushed
git log -S TEXT
Finds commits where TEXT was added or removed
Hunting down when something disappeared
git checkout SHA -- FILE
Restores one file as it was at a commit
Resurrecting the version that looked better
git cherry-pick SHA
Applies one commit's change to your current branch
Lifting a single good change out of an abandoned branch
When merge stops and asks you somethingIf two branches changed the same lines, git halts mid-merge and writes both versions into the file between markers that look like <<<<<<< and >>>>>>>. Nothing is broken and nothing is lost. Open the file, delete the markers and the version you don't want, save, then git add the file and git commit to finish the merge. If you would rather back out entirely, git merge --abort puts you back where you started. Conflicts show up more often in agent work, not less, because an agent branch and main routinely touch the same lines.
P.3 GitHub, remotes, and the pull request as a review surface
Git and GitHub are separate things, and conflating them causes a specific kind of confusion later. Git is the version control program running on your machine, and it works perfectly well with no internet connection and no account anywhere. GitHub is a company that hosts copies of git repositories and wraps them in collaboration features. The link between them is a remote, which is a named URL your repository knows about, conventionally called origin. You push commits up to it and pull other people's commits down from it, and git remote -v prints the URL so you can see whether you're talking to GitHub over HTTPS or over SSH.
A pull request proposes merging one branch into another and gives everyone a place to look at the change before it lands [17]. For solo work, this can feel like ceremony for an audience of one, and plenty of students skip it. Keep it, because the pull request has become the unit that AI coding runs in. Cloud agents deliver their work as one. GitHub's Copilot agent makes its changes on a branch and opens a pull request for you to review, and a Claude Code cloud session does the same, then watches the PR and pushes a fix when a check fails or a reviewer leaves a comment [28][29]. A pull request is also where you review generated code both ways at once. By function, since the checks that run on it (the tests, a build, a preview deployment) tell you whether it works, and by view, since it shows the whole change file by file, which is the fastest way to see what the agent actually touched compared with what its summary said. When you open a PR against your own repo and check it as though a stranger sent it, you're practicing the review the rest of the book asks for.
Reading a generated diff has its own order of operations. Begin with the file list, not the code, because the first tell that something went sideways is a file you didn't expect to change. Then slow down on the places where an agent takes the easy road: a package added to a lockfile to solve a problem four lines would have solved, a test quietly weakened so that a failing suite goes green, or a change buried in a config directory. And if a file that should have been ignored has appeared in the change list at all, stop there, because that's the subject of section P.6.
P.4 What an SSH key pair actually is
On August 13, 2021, GitHub stopped accepting account passwords for Git operations over HTTPS [9]. That deadline is worth remembering because it explains an error message thousands of people still hit: you type your GitHub password when git asks, and it's rejected even though the password is correct. Since that date, every authenticated operation runs on something token-shaped. A personal access token, an OAuth token issued by the GitHub CLI, a GitHub App installation token, or an SSH key pair. GitHub doesn't claim SSH is universally better than HTTPS, and its own documentation is careful about this: HTTPS works even from behind a firewall or proxy, while SSH connections are the ones a restrictive network is likely to refuse [7]. What SSH gives you is a credential you set up once and stop thinking about. Chapter 13 introduces that key from the credential-hygiene side, next to your API keys. Here we build one.
A key pair is two mathematically related files. Generating an ed25519 key produces a 32-byte secret stored in ~/.ssh/id_ed25519 and a public half derived from it, written into the same directory as ~/.ssh/id_ed25519.pub, with the derivation running one way only. You can compute the public half from the private half, and nobody can go backwards. Authentication works by challenge and response: GitHub holds only your public key, sends your client something to sign, and checks the signature. The private key never leaves your machine and never crosses the network, which is the whole reason this scheme is better than mailing a password to a server and hoping it stores it well.
The consequence students most often get backwards is which half is secret. The .pub file exists to be published. Pasting it into GitHub is its entire purpose, and there's nothing sensitive in it beyond the email comment at the end, whereas the other file is a bearer credential in the strictest sense. Anyone who obtains an unencrypted id_ed25519 can authenticate as you to every host that trusts it, silently, with no password prompt, no second factor, and no revocation list. Removing the public key from your GitHub settings page is the revocation, and generating a fresh pair is the rest of the fix.
Two practical tells keep you out of trouble. The public file is exactly one line and begins with ssh-ed25519. The private file is many lines and begins with a header announcing an OpenSSH private key. If something you're about to paste is more than one line or contains the word PRIVATE, stop and look again. The table below compares the credential types you'll actually choose between.
Credential
Scope
Good for
Watch out for
SSH user key
Every repo your account can reach
Your own laptop, set up once
Private key is a bearer credential with no second factor
HTTPS + fine-grained PAT
Chosen repos, chosen permissions
Scripts and CI where least privilege matters
Can be created with no expiry at all, so set one yourself and rotate on schedule
HTTPS + classic PAT
Broad scopes across everything you can access
The few gaps fine-grained tokens haven't closed
Over-broad by design, and GitHub deletes tokens left unused for a year
GitHub CLI / credential helper
Whatever the OAuth grant covers
Beginners, and machines you don't want to hand-manage
Falls back to a plaintext token in hosts.yml when no system keyring is available
Deploy key
Exactly one repository
A server or automation that needs one repo
Survives the creator losing repo access, and never expires
The two halves of an ed25519 key pair. Getting this backwards is the most common mistake in the chapter.
One line, or manyYour public key is a single line starting with ssh-ed25519. Your private key is many lines and says PRIVATE in the first one. Every accidental key leak we've seen in a classroom started with someone pasting the wrong file into a chat window because both filenames looked similar in a terminal listing.
P.5 Generating, adding, and verifying your key
Run ssh-keygen -t ed25519 -C followed by your email in quotes, accept the default file location, and choose a passphrase [1]. On macOS the pair lands in /Users/YOU/.ssh/, on Linux in /home/YOU/.ssh/, and in Git Bash on Windows at /c/Users/YOU/.ssh/. Use ed25519 instead of RSA [22][14] unless you're talking to something old enough to refuse it, in which case ssh-keygen -t rsa -b 4096 is the fallback. Ed25519 is worth preferring for concrete reasons: an ed25519 public key is about 68 characters long and carries roughly the security of a 3072-bit RSA key. Its signatures are also deterministic, which removes the whole family of attacks that has repeatedly broken implementations relying on a weak random number generator at signing time [27]. GitHub removed support for DSA keys entirely on March 15, 2022, and any RSA key uploaded to an account after November 2, 2021 works only with SHA-2 signature algorithms. One housekeeping detail while you're in the directory: OpenSSH refuses to use a private key that other users on the machine can read, so run chmod 700 ~/.ssh and chmod 600 ~/.ssh/id_ed25519 if it ever complains that your permissions are too open.
Next, get the key loaded into ssh-agent so you aren't retyping the passphrase every few minutes. That's the SSH key agent and not the coding agent, and the shared word trips people up constantly. Start it by running eval "$(ssh-agent -s)" [1]. The odd-looking wrapper is doing one job: ssh-agent prints a couple of settings when it starts, and eval applies them to your current shell so that later commands can find the running process. With the agent running, add the key. On macOS use ssh-add --apple-use-keychain ~/.ssh/id_ed25519, which stores the passphrase in your keychain, and on Linux and Windows plain ssh-add ~/.ssh/id_ed25519 is right. To make this stick across reboots on macOS, GitHub's documentation has you create ~/.ssh/config with a Host github.com block containing AddKeysToAgent yes, UseKeychain yes, and IdentityFile ~/.ssh/id_ed25519, and two caveats travel with that block. Omit the UseKeychain line if your key has no passphrase, and never copy it onto a Linux machine, where UseKeychain isn't a valid option and produces a bad configuration error on every ssh invocation.
Now upload the public half. Copy it with pbcopy < ~/.ssh/id_ed25519.pub on macOS, clip < ~/.ssh/id_ed25519.pub on Windows, or cat on Linux followed by selecting the output. In GitHub [2], go to your profile photo, then Settings, then Access, then SSH and GPG keys, then New SSH key. Give it a title naming the machine, leave the key type as authentication, paste, and save. The GitHub CLI does the same job with gh ssh-key add. Verify immediately by running ssh -T git@github.com. The first connection asks you to confirm GitHub's host key fingerprint, and the published ed25519 fingerprint to compare against is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU [5]. A successful run greets you by username and tells you GitHub doesn't provide shell access, which is the expected and correct outcome even though it looks like a rejection.
One more decision and one optional extra. The passphrase question is a genuine tradeoff [4]. Without one, anyone who gets a copy of your machine or its backups has working credentials to every host that trusts the key, and with one you pay a small recurring annoyance that ssh-agent and the macOS keychain mostly absorb. You can add a passphrase to an existing key without regenerating it using ssh-keygen -p -f ~/.ssh/id_ed25519. Separately, the same key can sign your commits if you re-upload it under the signing key type and set git config --global gpg.format ssh along with user.signingkey and commit.gpgsign true [26], noting the wart that the config keys still say gpg even in SSH mode. A Verified badge on GitHub means the signature matched a key on that account [18]. It says nothing about whether the code is any good.
ssh -T exits 1 even when it worksThe success message from ssh -T git@github.com comes back with exit status 1, because no shell session was opened. Students in scripted setups often treat that as a failure and start regenerating keys they didn't need to regenerate. Read the message, not the exit code.
P.6 Secrets, .gitignore, and the .env the model committed
Chapter 13 states the rule, and Chapter 7 explains why a model makes breaking it so easy. This section is about the other half, which is what you do once the rule has been broken, because with a model doing the typing that's a when and not an if. A .gitignore file lists patterns git should leave alone: node_modules, build output, editor cruft, and above all .env. Put one in every repository before the first commit, and start from the templates GitHub publishes for your language, since yours will be worse. The mechanic that catches people is that .gitignore only applies to files git isn't already tracking. If a secret file was committed before the rule existed, adding the rule changes nothing. Untrack it first with git rm --cached FILENAME. Personal preferences that shouldn't live in a shared repo, like your editor's scratch files, belong in a global ignore at ~/.config/git/ignore instead, where that list earns its keep twice over. Optional Chapter Q puts a dollar figure on what a node_modules listing or a build directory costs once an agent has read it into the context window, because everything in that window is re-sent and re-billed on every turn for the rest of the session.
GitHub has a second line of defense called push protection [11]. For pushes to public repositories, it's on by default for users, and it blocks the push outright when it detects a recognized provider credential, naming the file, the line, and the secret type, with a URL to bypass if you insist. Bypassing requires write access and a stated reason, and every bypass raises an alert and an audit log entry, though two limits matter more than the feature does. Coverage is by secret pattern, not by filename, so a homegrown key or a line like DB_PASSWORD=hunter2 may sail straight through. And repository-level push protection, the kind that also covers private repos, is off by default and requires a paid GitHub Secret Protection plan. Since May 5, 2026, secret scanning has also been generally available as a tool the GitHub MCP server exposes, which lets an agent in an MCP-connected editor scan a change for secrets before a commit is ever formed. That's a real improvement, and it still only fires if something calls it. Neither limit is a reason to skip these features. Both are reasons to stop reading a successful push as evidence that nothing sensitive left your machine.
When a secret does get committed, the instinct is to delete the file and move on. That instinct is wrong, and understanding why is the point of this section. Removing a file in a later commit doesn't remove it from the repository [23]. The old blob is still in the history and still reachable by its commit hash, and it remains reachable through cached views even after a force-push, and anyone who cloned or forked in the meantime has a copy regardless. Rewriting history with a tool like git filter-repo is worth doing afterward, and contacting GitHub Support to purge cached references is worth doing for a public repo, but neither of those is the first step, which is always to assume the credential is burned.
Rotate first, clean up secondIf an API key, token, or password reaches a commit that left your machine, treat it as public from that moment. Go to the provider, revoke the credential, issue a new one, and update wherever it was used. Only then worry about the git history. A rewritten history with a still-valid key in someone's fork is a leak you've documented rather than fixed.
P.7 Letting an agent use git
Agents are good at git and should be allowed to use it, with a boundary. The read-only and local-only commands are safe to auto-approve, because their worst outcome is noise: status, diff, log, add, commit, branch, and switch all stay inside your machine and inside a history you can inspect. The operations that reach the network or destroy work are the ones to gate behind your approval. Pushing, and especially git push --force, changes what other people see. History rewrites, hard resets that discard uncommitted work, tag and release creation, and anything touching remotes or credentials all belong in the ask-first column. This is the same permission boundary Chapter 3 describes, applied to the one tool an agent will reach for dozens of times a session.
Keep ~/.ssh outside what the agent can read. The private key has no second factor and no per-use prompt, so an agent that can cat it can also put it in a transcript, a log file, a bug report, a commit, or a request to a third-party service, all without anything looking obviously wrong at the time. The same argument covers ~/.aws/credentials, ~/.netrc, ~/.npmrc, ~/.docker/config.json, and ~/.config/gh/hosts.yml, which holds a GitHub CLI token in plaintext whenever the CLI can't reach a system keyring. Put those paths on the deny list in your tool's permission configuration, and keep a passphrase on your key so that copying the file alone doesn't immediately hand someone your account.
For anything automated, use an identity that isn't you. A fine-grained personal access token [8] scoped to one repository, a deploy key attached to a single repository [6], or a dedicated machine user account all limit what a compromised process can reach, and inside GitHub Actions the built-in GITHUB_TOKEN beats a token you pasted in. Deploy keys have one trap worth naming: they're tied to the repository and not to a person, so they keep working after the person who created them loses access. There's also a newer risk to hold in mind, and it's Chapter 14's prompt injection wearing a git costume. In May 2025, Invariant Labs showed that instructions planted in a public repository issue could steer an agent using the GitHub MCP server into copying private repository contents somewhere public. In July 2026, Noma Security published the same attack against GitHub Agentic Workflows, where an agent granted read access across an organization pulled a private repository's README into a public comment on the issue that told it to. An agent with repository access reads issue bodies, pull request descriptions, commit messages, and code comments, all of which are text someone else wrote. Anyone who can file an issue on your repo can put instructions in front of your agent.
Auto-approve
Ask first
Never unattended
git status, git diff, git log
git push
git push --force
git add, git commit
git merge into main
git filter-repo or any history rewrite
git branch, git switch -c
git reset --hard
Reading or copying ~/.ssh, ~/.aws, ~/.config/gh
git show, git blame
Creating tags and releases
Editing .gitignore to unignore a secrets file
Make the checkpoint the agent's jobPut a line in your rules file (CLAUDE.md or equivalent) telling the agent to commit the current state before beginning any multi-file change, with a message describing what it's about to attempt. You get the checkpoint without having to remember it, and the history reads like a log of what you asked for.
Summary
Version control stops being bookkeeping the moment a machine writes the code. This chapter walks the local loop (init, clone, status, add, commit, branch, merge, revert), then remotes and pull requests, then the practical centerpiece: SSH keys, which you need because GitHub turned off Git password authentication on August 13, 2021. The last third covers what happens when a model helpfully commits your .env, where the answer starts with rotating the credential and not with rewriting history. It closes on which git commands an agent may run unattended and why ~/.ssh belongs outside what it can read.
Key terms
Repository. A folder that git is tracking, marked by a hidden .git directory that holds the full history of every committed change.
Commit. A recorded snapshot of the tracked files at a moment in time, identified by a hash. In agent work, the checkpoint you can return to.
Working tree. The actual files in your folder right now. A clean working tree means nothing has changed since the last commit.
Diff. The line-by-line difference between two states of the code. The surface on which generated code is reviewed.
Branch. A named pointer into the history that lets a line of work proceed without touching the main one, so a bad session can be discarded whole.
Remote. A named URL, usually called origin, pointing at a hosted copy of the repository that you push to and pull from.
Pull request. A GitHub proposal to merge one branch into another, providing a readable diff and a place to comment before the change lands.
SSH key pair. Two related files: a private key that stays on your machine and a public key you publish. GitHub verifies you by asking your client to sign a challenge with the private half.
Public key. The half of an SSH key pair that exists to be published. One line that begins with ssh-ed25519 and is safe to paste into GitHub or anywhere else that needs to recognize you.
Private key. The half of an SSH key pair that never leaves your machine. Anyone holding an unencrypted copy can authenticate as you to every host that trusts it, with no second factor and no prompt.
Passphrase. A password encrypting the private key file on disk, so that copying the file alone isn't enough to use it. ssh-agent holds the decrypted key so you type it rarely.
ssh-agent. A background process that holds your decrypted private key in memory so a passphrase-protected key doesn't prompt you on every operation.
Deploy key. An SSH key attached to one repository instead of an account, used for servers and automation. It has no expiry and keeps working after its creator loses access.
Personal access token. A GitHub credential used in place of a password over HTTPS. Fine-grained tokens scope to chosen repositories and permissions; classic tokens carry broad scopes.
.gitignore. A file listing patterns git should leave alone (node_modules, build output, editor cruft, and above all .env), put in every repository before the first commit. It only applies to files git isn't already tracking; a secret committed before the rule existed needs git rm --cached to actually untrack it.
Push protection. A GitHub feature that blocks a push containing a recognized provider credential, naming the file and secret type. It matches patterns, not filenames, so homegrown secrets can slip through.
Credential rotation. Revoking a leaked secret at the provider and issuing a replacement. The first and only reliable response to a committed key.
Check your understanding
Q1. You're about to hand an agent a task that will touch several files. What does committing first actually buy you?
It causes the agent to write noticeably better code
A return point, and a diff showing only the agent's changes
It uploads your work to GitHub to serve as a backup
It stops the agent from editing any file you didn't name
Answer: B. The commit does nothing to the agent. It changes your position: you can reset back to a known state, and because the working tree was clean, everything git diff reports afterward is the agent's work. A commit is local, so it isn't a backup until you push it.
Q2. Which file do you paste into GitHub's SSH and GPG keys page, and why is that safe?
id_ed25519, because GitHub needs to sign challenges on your behalf
Either file, since both halves of the pair are equivalent
id_ed25519.pub, because the public half is meant to be published
A copy of id_ed25519 with its passphrase stripped out first
Answer: C. Only the .pub file goes to GitHub. It's one line, it's computed from the private key by a one-way derivation, and publishing it is its entire purpose. The private key never leaves your machine and never crosses the wire, because authentication works by signing a challenge locally.
Q3. An agent committed a .env containing a live Stripe key, and the push succeeded. What's the first thing to do?
Delete the file in a new commit and push again
Force-push a rewritten history so the commit disappears
Add .env to .gitignore so it won't happen again
Revoke the key at Stripe and issue a new one
Answer: D. Rotation comes first. The old blob stays reachable by commit hash, survives a force-push through cached views, and already exists in any clone or fork made in the meantime. Cleaning history and fixing .gitignore are both worth doing, but only after the credential itself is dead.
Q4. Git rejects your GitHub account password when pushing over HTTPS, even though the password is correct. Why?
GitHub dropped password auth for Git in 2021; use a token or key
Your account is rate-limited and will work again shortly
HTTPS remotes are no longer supported by GitHub at all
Your repository is private and private repos require SSH
Answer: A. Password authentication for Git operations was shut off on August 13, 2021. Every authenticated operation since then runs on a personal access token, an OAuth token from the GitHub CLI, a GitHub App installation token, or an SSH key pair. HTTPS still works fine, but the password field now wants a token.
Practice
Exercise 1. Generate an ed25519 key pair with a passphrase, load it into ssh-agent, add the public half to your GitHub account, and confirm with ssh -T git@github.com. Then take an existing repository with an HTTPS remote and re-point it at the SSH URL with git remote set-url, and push. Write down what the ssh -T output said and what its exit code was. (Hint: Compare the host-key fingerprint on first connection against GitHub's published list before typing yes. And the success message comes back with exit status 1, which is expected.)
Exercise 2. Run a controlled checkpoint experiment. Commit a clean state, give an agent a task big enough to touch three or more files, then read the full diff before accepting anything. Undo the run two different ways on two different branches: once with git reset --hard and once with git revert, and write two sentences on when each is the right choice. (Hint: The distinction is whether the commit has left your machine. Reset rewrites your local position; revert adds a new commit that undoes an old one, which is what you want for anything already pushed.)
Exercise 3. Audit a repository you already have for secret hygiene. Run git ls-files to list what's actually tracked, look for anything that shouldn't be there, and check whether your .gitignore rules are doing anything for files that were committed before the rules existed. Fix at least one real finding. (Hint: Ignore rules apply only to untracked files. If a file is already tracked, you need git rm --cached before the rule takes effect. Start from the template for your language in the github/gitignore repository rather than writing one from scratch.)
Exercise 4. Practice repository archaeology. In a repo with some history, deliberately overwrite something you like, commit the damage, then ask your agent to recover the earlier version by describing it in plain language ('the version of this component from before today') rather than naming a commit. Watch which commands it reaches for. (Hint: The agent will likely use git log -S or git log -p to locate the commit and git checkout SHA -- FILE to resurrect the file. The lesson is that you never needed to remember the SHA, but you did need the earlier version to have been committed at all.)
Sources
GitHub: Generating a new SSH key and adding it to the ssh-agent (https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent)
GitHub: Adding a new SSH key to your GitHub account (https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
GitHub: Testing your SSH connection (https://docs.github.com/en/authentication/connecting-to-github-with-ssh/testing-your-ssh-connection)
GitHub: Working with SSH key passphrases (https://docs.github.com/en/authentication/connecting-to-github-with-ssh/working-with-ssh-key-passphrases)
GitHub's published SSH key fingerprints (compare on first connection) (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints)
GitHub: Managing deploy keys, and why a machine user is often better (https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys)
GitHub: About authentication to GitHub (HTTPS vs SSH, tokens vs apps) (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-authentication-to-github)
GitHub: Managing your personal access tokens (fine-grained vs classic) (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)
GitHub changelog: Git password authentication is shutting down (Aug 12, 2021) (https://github.blog/changelog/2021-08-12-git-password-authentication-is-shutting-down/)
GitHub: Ignoring files, including untracking a file already committed (https://docs.github.com/en/get-started/git-basics/ignoring-files)
GitHub: About push protection for secret scanning (https://docs.github.com/en/code-security/secret-scanning/introduction/about-push-protection)
GitHub: Scanning for secrets with the GitHub MCP server (https://docs.github.com/en/code-security/how-tos/use-ghas-with-ai-coding-agents/scan-for-secrets-with-github-mcp-server)
GitHub changelog: Secret scanning with GitHub MCP Server is now generally available (May 5, 2026) (https://github.blog/changelog/2026-05-05-secret-scanning-with-github-mcp-server-is-now-generally-available/)
GitHub blog: Improving Git protocol security on GitHub (DSA and RSA/SHA-2 dates) (https://github.blog/security/application-security/improving-git-protocol-security-github/)
Noma Security: GitLost, tricking GitHub's AI agent into leaking private repos (July 2026) (https://noma.security/blog/gitlost-how-we-tricked-githubs-ai-agent-into-leaking-private-repos/)
GitHub: About pull requests (https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests)
GitHub: About commit signature verification (what Verified actually means) (https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification)
github/gitignore: official .gitignore templates by language (https://github.com/github/gitignore)
Pro Git, chapter 2: Recording changes to the repository (https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository)
Pro Git, chapter 7: Signing your work (https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work)
GitHub: Removing sensitive data from a repository (history persists, rotate the credential first) (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository)
Pro Git, chapter 2: Getting a Git repository (https://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository)
Pro Git, chapter 2: Undoing things (https://git-scm.com/book/en/v2/Git-Basics-Undoing-Things)
GitHub: Telling Git about your signing key (the SSH signing configuration) (https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key)
RFC 8032: Edwards-Curve Digital Signature Algorithm (deterministic signatures and the 128-bit security level) (https://www.rfc-editor.org/rfc/rfc8032)
GitHub documentation: about Copilot cloud agent, which makes its changes on a branch and creates a pull request for review (https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent)
Claude Code documentation: Claude Code on the web, including PR creation from a cloud session and auto-fixing a pull request on CI failures and review comments (https://code.claude.com/docs/en/claude-code-on-the-web)
Optional Q. The Token Economy: Working Frugally with AI
What a session actually costs, why turn 50 isn't the price of turn 1, and the habits that cut the bill.
Reference chapter. Look things up here as you need them; it isn't meant to be read straight through.
Most students discover the cost of AI coding the way people discover the cost of a phone plan, which is at the end of the month. This chapter makes the meter visible before it surprises you. The central fact, and almost nobody learns it on their own, is that a chat session re-sends the whole conversation as input on every turn, so a long session's cost grows with the square of its length. Once you see that, the habits that save real money stop being folklore and start being arithmetic. The goal is to know what you're spending and to spend it where it buys something. Minimizing the bill is a different goal, and a worse one.
By the end of this chapter, you will be able to
Explain what input tokens and output tokens are, what they cost, and why output is priced higher
Estimate the token volume of a piece of text or a source file and know which tools count tokens accurately
Show why cumulative session cost grows with the square of the number of turns, using a worked example
Describe what prompt caching does to that growth, what invalidates a cache, and how caching affects rate limits
Apply the frugality habits that save the most for the least effort: clearing, curating context, matching tier and effort, bounding agents, batching
Read your actual spend with /cost, /usage, /context, and the Console dashboards, and estimate a feature's cost before you build it
Tell metered billing from a subscription's rolling usage windows and diagnose whether a session, weekly, Opus-specific, or auto-compact message is the one you actually hit
Recognize when frugality becomes false economy and justify spending more on a single run
Q.1 What you're actually buying
Every request to a model bills two separate meters. Input tokens cover everything you send: the system prompt, your tool definitions, every message in the conversation so far, plus any file contents, images, or documents that came along for the ride. Output tokens cover everything the model generates, including its extended thinking, which is billed at the output rate even though you may never read it. On every current Claude model, the output rate is exactly five times the input rate, which is a clean ratio you can check against the price list. Output is the expensive direction, which is why skills like caveman (caveman.so) exist. It rewrites an agent's replies into terse fragments and claims to cut output tokens by about two thirds without touching the code it produces. As of the 2026-06-04 snapshot in Optional Chapter M, Opus 5 runs $5 per million input tokens and $25 per million output, Sonnet 5 runs $2 and $10, and Haiku 4.5 runs $1 and $5.
That five-times gap isn't arbitrary pricing. Input tokens are processed together in a single prefill pass, so a hundred thousand of them cost roughly one trip through the model. Output tokens are produced one at a time, and each one requires its own full forward pass through the network, so the price follows the compute. A practical consequence is that a verbose model, or a high effort setting that thinks at length before answering, hits the expensive meter, while a large pile of context hits the cheap one, so both matter, but they matter differently.
A token is roughly four characters or three quarters of a word in English, which is Anthropic's own published estimate and the number worth memorizing [4]. Inverted, a thousand words comes to about 1,333 tokens, a kilobyte of English prose comes to about 250 tokens, and a 500 KB plain-text file comes to about 125,000 tokens. File size on disk isn't a safe proxy for a PDF or a Word document, where fonts and embedded images dominate the bytes, so extract the text first or just count it, and two caveats keep that heuristic honest. First, the tokenizer changed with the current generation: every model from Opus 4.7 onward, which includes Opus 5, Fable 5, and Mythos 5, uses a newer tokenizer that produces roughly 30 percent more tokens for the same text than Sonnet 5 and earlier do, so any budget you calibrated on an older model is off by about a third, and your bill reflects the new counts. Second, and relatedly, tiktoken is the wrong tool for checking your math. That's OpenAI's tokenizer, and it materially undercounts Claude tokens. Anthropic ships a free token-counting endpoint that takes the same payload shape as a real request and returns the count, at no cost to call.
The unification worth carrying out of this section is that your bill and your context limit are the same tokens. Chapter 4 treated them as a ceiling, the thing you run out of, whereas here they're a price, the thing you pay for. A 2,000-line file you dumped into the window because it seemed easier than being specific costs you twice: once in dollars, and again in the space it occupies for the rest of the session.
Every request bills two meters. Output costs five times input on every current Claude model, because each output token needs its own pass through the network.
Measure instead of guessingThe count-tokens endpoint is free, and its limits are separate from message creation, so measuring costs you nothing and doesn't eat into the budget you need for real work. Because it counts under the tokenizer of whichever model id you pass, you can measure the tokenizer difference yourself by counting the same request twice under two model ids. Counting a source file against an equal-sized chunk of prose is a five-minute exercise that will recalibrate your intuition permanently.
Q.2 The quadratic surprise
The API is stateless. It doesn't remember your last message, which means the client has to send the entire conversation again on every single turn so the model can see what came before [2]. Anthropic's own documentation says it plainly: as a conversation advances, each user message and assistant response accumulates within the context window, and previous turns are preserved completely. Everything in the request counts, including the system prompt, every message, every tool result, and your tool definitions, so turn 20 of a chat doesn't bill for one message. It bills for turns 1 through 19 plus your new message, all over again, at full input price.
Work an example so the shape is undeniable. Say you're on Opus 5 at $5 per million input tokens [1]. Your fixed prefix, meaning the system prompt plus tool definitions plus your CLAUDE.md, comes to 10,000 tokens. Each question you type is about 200 tokens, and each reply is about 1,300, so every completed turn adds 1,500 tokens of permanent history, so the input billed on turn k is 10,200 plus 1,500 times (k minus 1), so turn 1 bills 10,200 input tokens, about five cents of input. Turn 20 bills 38,700, about nineteen cents. That's 3.8 times the input price of turn 1 for the exact same 200-token question. The table below adds the roughly three cents of output each turn also costs, which is why its per-turn totals run higher and its ratios look gentler.
Now sum it. Across N turns, the total input is the prefix times N plus 1,500 times N(N-1)/2, and that second term is the whole story, because it grows with the square of N, so twenty turns costs about $3.10 all in while one hundred turns costs about $45.48, meaning five times the turns produced 14.7 times the cost. Meanwhile, the student's mental model says they typed twenty questions of 200 tokens on top of a 10,000-token prefix, about 14,000 tokens, roughly seven cents, while the real input bill on the 20-turn session is 35 times that estimate. That gap is the linear intuition failing, and it widens the longer you sit there.
Agentic coding is worse than this toy model, and by a lot. Tool results are messages too. A file the agent read, a 400-line test failure, a grep that returned everything, a directory listing of a node_modules folder that should have been ignored: all of it lands in the transcript, and all of it is re-sent on every subsequent turn until the session is cleared or compacted. A single 2,000-line file read at turn 3 is still being paid for at turn 40. This is why the same feature can cost fifty cents or five dollars depending entirely on what was in the window while you built it.
Turn
Input tokens billed that turn
Cost of that turn (input + output)
Cumulative session cost
1
10,200
$0.08
$0.08
10
23,700
$0.15
$1.17
20
38,700
$0.23
$3.10
50
83,700
$0.45
$13.36
100
158,700
$0.83
$45.48
The expensive mistake is invisibleNothing in the interface tells you that turn 40 costs five times what turn 4 did. The session looks identical. This is the single most common cause of a bill that shocks someone at the end of the month, and the only cure is a habit, because the tool won't warn you.
Q.3 Prompt caching and what it changes
Prompt caching exists because that quadratic is a problem for the provider as much as for you. You mark a prefix of the request as cacheable, the API stores its processed form [3], and any later request whose prefix matches byte for byte reads the stored version instead of reprocessing it. The pricing is expressed as a multiplier on that model's base input rate. Writing to a cache with a five-minute lifetime costs 1.25 times base, writing to a one-hour cache costs 2 times base, and a read costs 0.1 times base. Anthropic states the break-even directly: caching pays off after just one cache read on the five-minute duration, or after two reads on the one-hour duration. That's a very low bar, which is why caching is on by default in most coding tools.
Run the same 20-turn session again with a cache breakpoint moved to the end of each new turn. The reads come to 450,300 tokens at $0.50 per million, about 22 cents. The writes come to 38,700 tokens at $6.25 per million, about 24 cents. Input drops from $2.45 to $0.47, roughly five times cheaper, and the all-in session cost drops from $3.10 to about $1.12, though the quadratic doesn't disappear so much as have its coefficient fall by about ten. That's the honest description, and it explains why long sessions became economically survivable without becoming economically fine. One detail catches people out: the cache lifetime refreshes for free every time the cached content is used, so an active session keeps rolling the five-minute window forward at no charge, but a coffee break longer than five minutes drops it, and your next turn pays a fresh write.
Caching is a strict prefix match over the rendered request, and the render order is tools, then system prompt, then messages [3], so a change at any level invalidates that level and everything after it, so changing a tool definition loses all three. Change the system prompt, and you keep the tools cache but lose system and messages. This is where the silent killers live, because they all sit early in the prefix and therefore poison everything downstream: a current-date line in the system prompt, a request id or UUID, a dictionary serialized without sorted keys so the field order wobbles, a tool list assembled per user. Switching models mid-conversation is the same category of mistake, because the model is part of the cache key, so a mid-session hop from Sonnet to Opus throws away the entire cache you had built.
Two failure modes are worth knowing about before you trust a cache. Prompts shorter than the model's minimum cacheable prefix simply don't cache, and no error is returned [3]. The minimum varies by model, and it tracks the model's generation rather than its price: 512 tokens on Opus 5, about a thousand on Sonnet 5, but four thousand on Haiku 4.5, so the budget model is ironically the hardest one to cache a short prompt for. Rather than trusting a printed number, verify empirically: send the same prefix twice and check that cache_read_input_tokens is nonzero on the second call, where a persistent zero means something upstream is invalidating. Cache reads don't count toward your input-tokens-per-minute rate limit on current models. In the 20-turn example, the input measured against that limit falls from 489,000 tokens to 38,700, which is 12.6 times more effective throughput on the same tier, so caching buys you headroom rather than only a discount.
Operation
Multiplier on base input
Opus 5 $/1M
Lifetime
Uncached input
1x
$5.00
n/a
Cache write, 5 minute
1.25x
$6.25
5 min, refreshed free on each hit
Cache write, 1 hour
2x
$10.00
1 hour
Cache read (hit)
0.1x
$0.50
same as the preceding write
Q.4 The habits that actually save money
The habit that saves the most money is the dullest one: clear the session between unrelated tasks. That sounds like tidiness advice, but it follows directly from the arithmetic in Q.2. Every stale token in the window is re-billed on every message you send for the rest of the session, so the transcript of the bug you fixed an hour ago is charged again while you work on something entirely different. Anthropic's own diagnosis of unexpectedly high Claude Code spend is blunt about it: surprise bills usually trace back to long sessions that were never cleared, or to Opus left as the default model. Clearing doesn't destroy the thread, because Claude Code persists past sessions and you can come back to one with the resume flag if it turns out to matter. When a session genuinely needs to stay alive, compact it with an instruction instead of blindly, telling it what to preserve, so the summary keeps the code samples and drops the exploratory chatter. A longer-running agent has a blunter option than compaction, which is context editing, deleting old tool results and thinking blocks outright on the reasoning that a directory listing from forty steps ago has no remaining value at all.
The second habit is curating what enters the window in the first place. A specific request, like adding input validation to the login function in a named file, doesn't trigger the broad codebase scan that a request to improve this codebase does. Naming the file matters more than it looks: an agent left to find things greps and reads its way across the repo, and every one of those search-and-read passes is input tokens for material that mostly won't survive into the answer. The rules file cuts both ways. A good CLAUDE.md spares the agent a re-derivation of your project, but it loads at session start and is re-billed on every turn, which is why Section 3.3 caps its length. Workflow-specific instructions belong in skills that load on demand, not in the file that's always present, and the same logic applies to the control surface Chapter 3 set up. Disable MCP servers you aren't using, and keep generated output and third-party library code behind ignore rules. Optional Chapter P has you build that same list for version control, and the overlap is close to total, because a directory you would never commit is almost never one you want an agent reading into the window either. The context command shows you what's actually occupying space, which beats guessing. A hook that greps a 10,000-line log for errors before the model ever sees it turns tens of thousands of tokens into hundreds.
The third group is about matching the spend to the shape of the job. Tier matching is familiar from Chapter 1, with Sonnet at 40 percent of the Opus input price and Haiku at 20 percent [1]. Effort is the lever most students never touch [10], and it hits the expensive meter, because thinking tokens bill as output at five times the input rate. Optional Chapter M gave you the ladder, low through medium, high, xhigh, and max, as a quality decision, where here it's a price. Low effort suits scoped, latency-sensitive work, and the documented advice is to start at the default of high, step up to xhigh for demanding coding and agentic runs, and step down to medium or low as your primary cost control wherever your own checks show that quality holds. Raising effort raises token spend per turn, so the case for paying it is the one Q.6 makes, which is that a run that lands on the first try beats a cheaper run that needs a second pass. Chapter 8 already gave you the step limit as a safety guardrail against a runaway loop. Read it again as a cost control, because every failed iteration's output becomes the next iteration's input, and a budget the agent can see and pace itself against does something a hard cap can't: it lets the agent choose to finish cheaply instead of being cut off mid-thought. The table below covers the rest of this group, including the flat 50 percent the Batch API takes off both meters for anything that doesn't need to be interactive.
The structural moves matter as much as the per-turn habits. Delegating verbose operations to a subagent keeps the noisy output, such as full test logs or fetched documentation, inside the subagent's own context [15], so only a summary returns to your main conversation, and only the summary gets re-billed forever after. The counterpoint is that a team of agents burns far more tokens than a single session; Section 3.7 has the multiple, and it's steep enough that running them in parallel is a choice worth making on purpose. The last habit is to plan before you implement. Discovering a wrong direction at turn 2 costs almost nothing, and discovering it at turn 30, after 300,000 accumulated input tokens, costs what the table in Q.2 says it costs, so rolling back is usually cheaper than arguing forward.
Habit
Why it works
Rough impact
Clear between unrelated tasks
Stale context is re-billed on every later message
Largest single lever; attacks the quadratic directly
Keep prompt caching intact
Cache reads bill at 0.1x and skip the rate limit
About 5x cheaper input on a long session
Trim the rules file and ignore junk
Anything loaded at start is re-billed every turn
Proportional to what you cut, on every turn
Batch related asks into one turn
Pays the accumulated prefix once instead of three times
Meaningful on chatty back-and-forth work
Match model tier to the task
Sonnet is 40 percent of Opus input, Haiku is 20 percent
Large, but see Q.6 before over-applying
Tune effort to the task
Thinking bills as output at 5x the input rate
Large on reasoning-heavy turns, both directions
Bound agent loops with a budget
A stuck loop feeds its own output back as input
Prevents the worst case rather than trimming the average
Use the Batch API for non-interactive work
Flat 50 percent off input and output, stacks with caching
Halves the bill on evals and bulk jobs
The frugality habits, ordered by savings per unit of effort. The top rung costs nothing but a change of reflex.
One command, most of the savingsIf you adopt exactly one habit from this chapter, make it clearing the session when you switch tasks. It takes a second, it's reversible because past sessions can be resumed, and it removes the compounding term from your bill.
Q.5 Subscriptions, meters, and knowing your number
There are two billing shapes, they fail in different ways, and matching the shape to the work is itself a frugality decision. Heavy interactive coding is exactly the usage pattern a flat-rate subscription absorbs [13]: a developer deep in a project can burn thousands of dollars of metered tokens in a few weeks producing output a fixed monthly plan would have covered. Metered billing earns its place where flat-rate can't go, in your own product's inference, in batch jobs, in anything programmatic. The expensive mistake is running your personal coding meter-first because the credits happened to be there. Metered billing through the Console charges per token at the published rates, with an organization-level monthly spend cap that rises as your usage tier advances, from $500 at the entry tier to $1,000 at the middle tier and $200,000 at the top published tier, plus a lower self-imposed cap you can set yourself. Those ceilings move, so read them off the Console limits page, not off this paragraph. Metered access also carries per-model rate limits measured in requests, input tokens, and output tokens per minute, and exceeding any of them returns an HTTP 429 with a retry-after header, while subscription billing works differently. As of mid-2026, Claude Pro is $20 a month, Max is $100 or $200 depending on the multiplier, and Team seats are priced per person, and subscription limits aren't dollar-denominated at all. They're rolling usage windows: a five-hour session window plus two weekly limits, one spanning all models and one specific to Opus, resetting at a fixed time tied to your account.
That difference explains a set of messages students routinely misread. If you're told you hit your session limit or your weekly limit, that's the seat-based window [14], and switching models won't restore access because the window spans all models. If you're told you hit your Opus limit specifically, that one is model-scoped, and dropping to Sonnet does keep you working. A context or auto-compact warning is a third thing entirely and isn't a usage limit at all: it means the conversation approached the model's input ceiling and older history is being summarized. Anthropic deliberately doesn't publish message or token counts for the five-hour and weekly windows, so any precise figure you find in a forum thread is unofficial, so learn the structure and check Settings rather than the number.
You can't manage a number you've never seen, so go look. Inside Claude Code, the usage command, which the cost command is now an alias for, opens one screen with two halves. The session block at the top prints what the current session has spent, broken out by model, with a dollar figure the tool computes locally from list rates [11][12]. Below it, on a paid plan, sit the bars showing where you stand against your five-hour and weekly windows. Reading the wrong half is easy, and the whole point of this section is that the two halves answer different questions: one is what this conversation cost, and the other is how much of your plan is left. Neither is the same as the per-category token breakout, which comes from the Console usage dashboard, from the telemetry Claude Code can export, or from the usage object on a raw API response. The shape of that breakout is itself instructive: a healthy session shows something like a thousand uncached input tokens against nearly a million cache reads, which tells you caching is working. The context command shows what's currently occupying the window, which is where you find the file nobody needed. For authoritative billing, plus per-model rate-limit and cache-hit-rate charts, the Console usage dashboard is the source of truth. If you're calling the API directly, every response carries a usage object, and your true input is the sum of three fields, not one: uncached input tokens, cache creation tokens, and cache read tokens. Reading only the first field is how people convince themselves their spend is a tenth of what it is.
Estimating a feature before you build it is a five-minute exercise once you have the formula. Note which meter this is. Chapter 13 estimates what a shipped feature costs per user in production, forever. This estimates what building it costs you once, in your own session, and the two numbers have almost nothing to do with each other. Count your fixed prefix with the token-counting endpoint. Guess the number of turns honestly, which means higher than feels comfortable. Then apply the sum from Q.2 and multiply by your model's rates. For calibration against reality, Anthropic's published figures for Claude Code put the average around $13 per developer per active day, with 90 percent of users staying under $30 per active day and team spend at roughly $150 to $250 per developer per month. Idle background usage sits under four cents a session in their published numbers. Those figures date quickly and depend heavily on which model is the default, so treat them as an order of magnitude. If your own number is several times those benchmarks, blame session hygiene before you blame the difficulty of your work. Other tools price differently again, bundling a prepaid pool of metered usage into a monthly fee, which behaves less like a rolling window and more like a phone plan with overage. Those terms have changed repeatedly, so confirm them on the vendor's own pricing page rather than trusting a figure printed in a book.
Three different messages, three different meaningsA session or weekly limit spans all models, so switching models won't help. An Opus limit is model-specific, so switching to Sonnet will. An auto-compact or context warning isn't a usage limit at all. It's the conversation approaching the model's input ceiling. Diagnosing which one you hit saves you from applying the wrong fix.
Q.6 When frugality is false economy
Everything above pushes in one direction, so this section pushes back, and it deserves the same weight as the tactics. The cheap-model reflex often loses on pure token arithmetic before you count anything else. Suppose a task takes the same token volume regardless of model. Three failed Sonnet 5 attempts bill $6 per million on input against Opus 5's $5, and $30 per million on output against Opus's $25. Per unit of work, the cheap route already costs about 20 percent more on both meters, and it costs more than that once you count that retries inside a live session carry the failed attempt's transcript forward, so the second and third attempts each bill a larger context than the first did. Re-rolling the same mid-tier model is one of the most reliably expensive things a student does while believing they're saving money.
Chapter 1 told you to start cheap and escalate. This is the amendment: start cheap on the parts of the job that are cheap to check, not on the job as a whole. Against Haiku the arithmetic genuinely flips, with three attempts at $1 per million input beating one at $5. That's exactly why the pattern that works is triage with a cheap model and solve with a capable one, instead of re-rolling a mid-tier model until it happens to land. Have Haiku find the file, summarize the log, or classify which of four categories the bug falls into, then hand the actual reasoning to Opus. One caution on how you route: switching models mid-conversation invalidates the entire prompt cache, so make the switch at a task boundary or inside a subagent rather than halfway through a live thread.
Then there's the term that dominates every other term, which is your time. The 20-turn session in Q.2 cost $3.10. If a cheaper run fails and you spend an hour untangling what it did, then at any hourly rate a student might plausibly assign to their own time, the repair costs several times the entire session. This holds even more sharply for professionals, where an afternoon of debugging a bad architectural choice buys nothing and costs a day. Trading three dollars for an afternoon is a bad trade dressed as discipline. The same reasoning is why planning before implementing is a cost control and not a formality.
The rule we want you to leave with is to spend deliberately, not minimally. Before a run, three questions cover most cases. How bad is it if this is wrong, meaning what has to be undone. How quickly will you know it's wrong, since a task with a fast, cheap verification tolerates a cheap model far better than one where the failure surfaces two weeks later. How much of your own time sits on the other side of the trade. Answer those honestly, and you'll sometimes reach for Haiku on a rename and sometimes open Opus at max effort on a schema design, which is the correct outcome. The table below prices the same feature four ways, and the spread between the best and worst rows is 6.5 times for identical work.
How the work was done
Input billed
Input cost
Output cost
Total
Opus 5, fresh cleared session, no caching
343,200 tokens
$1.72
$0.52
$2.24
Opus 5, fresh cleared session, caching on
32,700 written + 310,500 read
$0.36
$0.52
$0.88
Opus 5, appended to a 29-turn session never cleared
1,039,200 tokens
$5.20
$0.52
$5.72
Sonnet 5 stalls after 16 turns, then Opus 5 redoes it
343,200 + 343,200 tokens
$2.40
$0.73
$3.13 plus your afternoon
The cheapest row isn't the cheapest modelNotice which row wins. The winner is the expensive model run in a clean, well-cached session, not the smaller model. Session hygiene beats tier-downgrading on this task, and it does so without costing you any quality.
Summary
Every request bills two meters: input tokens for everything you send and output tokens for everything the model generates, at five times the input rate. The fact that reorganizes everything else is that the API is stateless, so each message re-sends the entire conversation as input, which is why a 20-turn Opus 5 session costs about $3.10 while a 100-turn one costs about $45.48. Prompt caching cuts the input side of that by roughly ten times and stops cached reads from counting against your rate limit, and the habits that follow all attack the same squared term, the largest of them being to clear between unrelated tasks. The chapter closes on reading your own numbers and on the counterweight the tactics need, which is that three failed cheap attempts plus an hour of repair is worse value than one capable run that lands.
Key terms
Input tokens. Everything you send in a request: system prompt, tool definitions, the entire message history, file contents, images, and documents, billed at the model's lower rate.
Output tokens. Everything the model generates, including extended thinking you never read. Billed at exactly five times the input rate on every current Claude model, because each output token needs its own forward pass.
Quadratic cost growth. The pattern where per-turn cost rises linearly with turn count and cumulative session cost rises with its square, because a stateless API re-sends the whole conversation as input on every turn.
Prompt caching. Marking a prefix of a request so the API stores its processed form. Matching later requests read the stored prefix at 0.1x the base input price instead of reprocessing it at full price.
Cache write vs. cache read. A write stores a prefix and costs 1.25x base input for a five-minute lifetime or 2x for an hour, while a read hits an existing prefix and costs 0.1x, so caching pays off after a single read on the five-minute duration.
Cache invalidation. Losing a cached prefix because the request no longer matches byte for byte. Common causes are a date or UUID in the system prompt, a reordered tool list, and switching models mid-conversation.
Compaction. Summarizing earlier turns so a long session fits and stops re-billing the full transcript. Compacting with an explicit instruction preserves what you care about instead of leaving the choice to chance.
Context editing. Clearing old tool results or thinking blocks outright instead of summarizing them, which suits long-running agents whose old tool output has no further value.
Context hygiene. The habit of keeping only what the current task needs in the window: clearing between unrelated tasks, compacting with an instruction, and keeping generated output and unused tools out of the prefix.
Effort. A setting from low through medium, high, xhigh, and max that controls how much the model thinks before answering. Optional Chapter M treats it as a quality decision; here it's a price, because thinking tokens bill at the output rate.
Skill. A block of workflow-specific instructions your tool loads only when a task calls for it, instead of on every turn the way a rules file does. Moving rarely relevant instructions into skills shrinks the prefix you pay for on every message.
Hook. A small program your tool runs automatically at a fixed point, such as before a command's output reaches the model. A hook that filters a log down to its error lines converts tens of thousands of tokens into hundreds.
Subagent. A separate model instance your main session delegates a noisy job to. Its transcript stays in its own window and only a summary returns, so only the summary is re-billed for the rest of your session.
Task budget. A token ceiling an agent can see and pace itself against, distinct from a hard maximum-tokens cap the model can't see. It bounds the worst case of a stuck loop.
Batch API. A non-interactive endpoint that runs requests asynchronously at a flat 50 percent discount on both input and output, stacking with prompt caching, which suits evals, bulk classification, and document processing.
Metered billing. Paying per token at published rates, with organization spend caps and per-minute rate limits on requests and tokens.
Rolling usage window. The subscription model's limit structure: a five-hour session window plus two weekly limits, one across all models and one model-specific, rather than a dollar balance.
False economy. A saving that costs more than it saves. The canonical case is three failed cheap attempts plus an hour of repair against one capable run that lands.
Check your understanding
Q1. You're 20 turns into a chat session. Compared with turn 1, roughly what does turn 20 cost, assuming your question is the same length both times?
The same, since you sent an identical number of tokens
Several times more, since the whole conversation is re-sent each turn
About the same plus a little, since only your new message is new
Less, since the model has already loaded all of that context
Answer: B. The API is stateless, so the client re-sends the whole conversation on every request. With a 10,000-token prefix and 1,500 tokens added per turn, turn 20 bills 38,700 input tokens against turn 1's 10,200, roughly 3.8 times the cost for an identical question.
Q2. A 100-turn session costs about $45.48 on the chapter's Opus 5 model. A 20-turn session on the same assumptions costs about $3.10. Why is the ratio 14.7x rather than 5x?
Output tokens are billed at five times the going input rate
Longer sessions switch to a more expensive model automatically
Cumulative input cost grows with the square of the turn count
Rate limiting adds a surcharge past a certain session length
Answer: C. Total input across N turns is the prefix times N plus the per-turn history times N(N-1)/2. That second term dominates as N grows, so cumulative cost scales with the square of the turn count, not linearly.
Q3. Which change is most likely to silently destroy your prompt cache on every request?
Asking a noticeably longer question than usual
Reading one large file late in the session
Raising the effort level from high up to xhigh
Putting the current date and time in the system prompt
Answer: D. Caching is a strict prefix match, and the system prompt renders before the messages. A timestamp that changes on every request invalidates the system cache and everything after it, with no error to tell you.
Q4. A task needs about the same token volume on any model. Three Sonnet 5 attempts fail before you switch to Opus 5, which succeeds in one. What does the arithmetic say about the cheap-first strategy here?
It cost more before counting repair time, and retries billed more context
It saved money, since Sonnet is 40 percent of the Opus input price
It's a wash, since the total token volume ends up identical
It saved money on input but not on output
Answer: A. Three Sonnet attempts bill 3 x $2 on input against Opus's $5, and 3 x $10 on output against $25. The cheap route already loses, and it loses by more once you count that each retry carried the failed transcript forward as input. Against Haiku the arithmetic flips, which is why triage-cheap and solve-capable is the pattern that works.
Practice
Exercise 1. Take a real session you ran this week. Estimate its total input tokens using the formula from Q.2: your fixed prefix times the number of turns, plus the per-turn history times N(N-1)/2. Then run the cost command in your tool and compare your estimate to the actual number. Write two sentences on where you were wrong and why. (Hint: Most people underestimate the per-turn history badly, because tool results and file reads are much larger than the messages they typed. Check the context command to see what's actually in the window.)
Exercise 2. Count the tokens in one 200-line source file from your project and in a piece of English prose of the same byte size, using the free count-tokens endpoint. Then count the same source file again under claude-haiku-4-5 instead of claude-opus-5, which is the accessible pair that straddles the tokenizer change. Report the two ratios you found. (Hint: The prose-versus-code comparison tests the four-characters-per-token heuristic on your own material. The two-model comparison should surface whatever tokenizer difference exists between the two models you picked; report the ratio you actually measure, not the one printed in Q.1. If you haven't called an API before, ask your coding tool to make the two count-tokens calls for you and print the numbers. Reading the result is the exercise, not writing the request.)
Exercise 3. Find one thing loaded into every session you run that doesn't earn its keep: an over-long rules file, an MCP server you never call, a directory that should be in an ignore file. Remove it, then measure the fixed prefix before and after and multiply the difference by a typical session length of 30 turns. (Hint: The context command will show you what's occupying the window. Anything in the fixed prefix is billed once per turn, so a 1,200-token cut over 30 turns is 36,000 tokens per session, every session.)
Sources
Claude API pricing: per-model input, output, and cache rates (https://platform.claude.com/docs/en/about-claude/pricing)
Context windows: why every turn re-sends the full conversation (https://platform.claude.com/docs/en/build-with-claude/context-windows)
Prompt caching: multipliers, lifetimes, prefix matching, and invalidation (https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
Token counting: the free count-tokens endpoint and the 4-characters heuristic (https://platform.claude.com/docs/en/build-with-claude/token-counting)
Messages count-tokens API reference (https://platform.claude.com/docs/en/api/messages/count_tokens)
Rate limits: RPM, ITPM, OTPM, and why cache reads do not count (https://platform.claude.com/docs/en/api/rate-limits)
Batch processing: the flat 50 percent discount for non-interactive work (https://platform.claude.com/docs/en/build-with-claude/batch-processing)
Compaction for long-running agents (https://platform.claude.com/docs/en/build-with-claude/compaction)
Context editing: clearing old tool results and thinking blocks (https://platform.claude.com/docs/en/build-with-claude/context-editing)
The effort parameter and its cost implications (https://platform.claude.com/docs/en/build-with-claude/effort)
Claude Code: manage costs effectively (https://code.claude.com/docs/en/costs)
Claude Code: monitoring usage and OpenTelemetry metrics (https://code.claude.com/docs/en/monitoring-usage)
Claude plans and pricing (Free, Pro, Max, Team, Enterprise) (https://claude.com/pricing)
Claude usage limits: the five-hour session window, the weekly limit, and the Opus limit (help center) (https://support.claude.com/en/articles/9797557-usage-limit-best-practices)
How we built our multi-agent research system (token economics of agent teams) (https://www.anthropic.com/engineering/multi-agent-research-system)
Console usage and cost dashboard: authoritative billing and cache-hit-rate charts (https://platform.claude.com/settings/usage)