Everyone’s Optimizing the Wrong Thing
Spend an hour reading about AI coding and you’ll come away thinking the whole game is the prompt. The perfect wording. The magic phrasing. The newest, biggest model. Get those right, the story goes, and the code writes itself.
I run an AI agent every day on a system real people depend on, and I’ve come to believe almost the opposite. The prompt matters least. The model matters less than you think. What actually separates an agent that ships reliable software from one that produces confident nonsense is the unglamorous stuff around the agent — the memory, the guardrails, the codified procedures. The infrastructure.
This is the least exciting claim I’ll make, and I’d argue the most useful: an AI agent is only as good as the context and the constraints you give it. Prompt cleverness is a rounding error next to that. Let me show you what the infrastructure actually is.
An Agent With No Memory Relearns Everything, Badly
Start with the problem the infrastructure exists to solve. A fresh AI session knows nothing about your project, your standards, your past decisions, or the three mistakes it made last week. Left to guess, it guesses — plausibly, and often wrong, in exactly the way I described in From Vibe Coding to AI Engineering. Every session becomes day one.
So the first piece of infrastructure is a project rules file the agent reads at the start of every session — the standing instructions that make it reproducible instead of improvisational. Not prompt tricks. Durable rules: how we name things, the boundaries the agent never crosses, the checklist it runs before proposing a change, where the canonical utilities live. Adding a rule here is how you lock in a good pattern permanently. Deleting one is how you let it go. The agent stops rediscovering your conventions and starts respecting them.
The second piece is memory that persists across sessions — short, topic-keyed notes the agent reads when it starts: who I am and how I like to work, corrections it’s been given before (so it stops repeating a mistake), the non-obvious state of the project. This is the difference between an assistant that knows you and one that reintroduces itself every morning. Crucially, memory is for what can’t be derived from the code — the judgment and the context — not for things the codebase already says.
Whose Rules? Not All Infrastructure Belongs to Everyone
There’s a distinction I just glossed over, and it matters more than any single rule: this infrastructure isn’t one undifferentiated pile. It has scope — and the scope is the point.
Some of it belongs to the project: the conventions, the boundaries, the canonical helpers — the shared truth about this codebase that everyone who touches it inherits. Write it once, and every engineer and every session starts on the same page.
But some of it belongs to the person. Each of us brings a unique blend of knowledge, taste, and hard-won judgment to the work, and it shows up in how we brief an agent, what we tell it to watch for, the corrections we’ve each accumulated over years. That layer is personal, and it should stay personal — it travels with the engineer from project to project. The point of all this scaffolding was never to flatten a team into interchangeable operators running identical rules; that would erase the exact expertise that makes the team worth anything in the first place. Good infrastructure amplifies each person’s blend. It doesn’t average it away.
Between the two sits the layer worth encapsulating into a shared skill or process — a review pass, a way of scoping a ticket — so a new teammate inherits the team’s hard-won practice on day one instead of absorbing it by osmosis over a month. That’s the real onboarding accelerant: hand the newcomer the commons fast, while still leaving room for their own personal layer to grow, because their judgment is the thing you hired them for.
So it’s three layers, not one: the project’s shared truth, the person’s individual expertise, and the portable commons that carries practice between them. Keep them distinct, and each does its job. Blur them — try to standardize the personal layer, or trap the common one in a single person’s head — and you either flatten your people or fail to onboard them.
The Boring Catalog That Prevents Half the Bugs
Here’s my favorite piece of infrastructure, because it’s so mundane and it prevents a whole class of failure.
When an agent needs a function — to format a date, to check a permission, to render a chart — it will happily write a brand-new one. Do that a hundred times and you get a hundred near-duplicate helpers that slowly drift apart, and the drift is where cosmetic bugs breed. Two date formatters that agree today and disagree after a daylight-saving change. Two permission checks that used to match.
The fix is a catalog of the canonical helpers — an index the agent (and every human) is told to scan before writing new logic. Reuse the one that exists; don’t mint a rival. It sounds like bureaucracy. It’s actually the single highest-leverage document we keep, because it turns “the agent reinvented the wheel, and the wheels don’t match” from a recurring surprise into a non-event.
Automate the Rules a Human Would Skip
The next principle is a quiet one: every rule you can enforce mechanically is a rule nobody — human or agent — can accidentally skip.
If a check can run without anyone thinking about it — a test suite, a lint, a scan for a known bad pattern, a formatting gate — then it should, and then your scarce human attention (and your scarce AI tokens) get spent on judgment instead of on remembering a checklist. Automation earns its keep here in a specific way: it pays for itself the first time it catches the thing everyone forgot to look for. The goal isn’t to automate the thinking. It’s to automate everything that isn’t thinking, so the thinking gets the attention.
Codify the Procedures, Don’t Re-Explain Them
The last layer is procedure. The valuable, repeatable workflows — how we run a full review, how we prepare a change to ship — shouldn’t live in someone’s head to be re-explained each time. They get codified as named commands the agent can invoke: run the review pass, run the ship checklist. The multi-round review I described in Review Is the Product is one of these — a procedure invoked by name, not reconstructed from memory every time. Codifying it means it runs the same way on a tired Friday as on a fresh Monday, which is the entire point of a procedure.
And you should watch your own practice with the same evidence-first eye you bring to the code. Instrument how the agent is actually being used — where the time goes, which patterns recur — so you’re improving the workflow on data, not on vibes. (Claude Code’s /insights is one way in — a pass over how you’ve actually been working that tells you what’s going well, what to fix, and what’s worth turning into a reusable skill; the principle is what matters — close the loop on your own process.)
Why This Is the Human Part, Not the Machine Part
It would be easy to read all of this as “make the AI more autonomous.” It’s the reverse. Every piece of infrastructure I’ve described is a way of encoding human judgment into the environment so that the human stays in control even as the agent moves fast.
The rules file is our judgment about what “good” means here. The memory is our accumulated corrections. The catalog is our decision about the right way to do a common thing. The automated checks are our standards, made unskippable. None of it makes the agent think for us. All of it makes the agent act within the boundaries we thought about carefully, once. That’s what lets you hand more of the typing to a machine without handing over the wheel.
The teams that ship reliably with AI aren’t winning on prompts or models. They’re winning on context engineering — the patient work of giving the agent a memory, a rulebook, a catalog, and a set of guardrails. It’s not glamorous, and it doesn’t demo well; it’s just what makes the speed safe. A well-equipped agent on an average model beats a bare agent on the best model every time — because the average one is operating inside a structure a human built on purpose, and the bare one is guessing.
Stop tuning the prompt. Build the memory, the rules, the catalog, and the guardrails. The infrastructure is the real work — the agent is just the part that moves.