What a year of AI agents taught me that the demos left out

I have sat through enough agent demos this year to recite the beat: a clean repo, a ticket that was written for the camera, a green check, a smile at the audience. Then the lights come up and someone asks what happens when the ticket is a lie, the repo is a decade old, and the check is green because the test mocks the only thing that matters. The demo does not have that slide.

I used to lead platform teams. I still spend my weeks on agentic design — the shift from software that does what you typed to software that attempts a goal. A year of that, in real services, taught me a short list of things the keynote never had time for. This is not a recap of the habits that survived. That piece is the workflow. This one is the omissions.

The demo is a product. Your repo is not

Every polished agent recording I watched this year had three hidden props. The task was specified by someone who already knew the answer. The repository had been tidied, or was new. The evaluation was a unit test that the demonstrator wrote five minutes earlier. Remove any one of those and the magic number — “it one-shotted the ticket” — collapses into a work session that looks like 2024 with better autocomplete.

I stopped treating “it worked on stage” as information. I started asking: who wrote the ticket, how old is the module, and what would still pass if the implementation were theater. If the vendor cannot answer those without switching to a second laptop, they showed you a commercial.

Context is a leaking bucket

Demos imply that a bigger window is a better engineer. In a real service, the relevant facts are not the last N tokens. They are a comment in a PR from 2022, a dashboard that only exists in Grafana, a Slack thread where someone said “never retry this endpoint,” and a customer who is an exception in a spreadsheet. Agents do not have that. They have what you stuffed in, what they retrieved, and what they invented to fill the join.

The omission is not that retrieval is imperfect. Everyone knows retrieval is imperfect. The omission is how often the agent is sure after a bad retrieve. I watched a tool “fix” a rate limiter by deleting it, because the code it could see looked unused and the incident doc it could not see said the limiter was the only reason the vendor did not ban us. A human who had been paged would have hesitated. The agent did not hesitate. Hesitation is not in the demo because hesitation looks like slowness.

What I want now, and what almost no demo shows, is a first-class “I might be missing a file” state. Not a longer chain. A stop.

Permissions are the product, and they are ugly

On stage, the agent has the keys. In a company, the agent should not have the keys. That sentence is obvious and still missing from the reel, because a demo that asks for approval every time a file changes is a demo that looks broken.

We spent months arguing about blast radius. Can it run tests? Yes. Can it push? No. Can it talk to staging? Sometimes. Can it rotate a secret? Absolutely not. Can it open a PR? Yes, if a human still clicks merge. Those rules are the actual design work. The model is a component. The permission graph is the product. Vendors sell the component. You live in the graph.

I also learned that “the agent asked me first” is not a control if the ask is a wall of diffs at 6 p.m. Approval fatigue is how bad changes land. Demos never show a tired approver. They show a fresh one, nodding.

Cost is not a footnote

A five-minute recording does not include the invoice. A year does. We had weeks where agent spend was a rounding error and weeks where a looping “just one more try” on a stubborn type error cost more than the engineer’s time to do the edit. The demo culture treats tokens as weather. Finance treats them as a line.

The leftover lesson is meaner: the agent will not tell you when it should stop. It does not have a sense of diminishing returns. You have to put a budget on the task — minutes, dollars, retries — the way you put a budget on a cloud job. I started writing the budget in the ticket. It felt petty. It cut the stupid loops in half.

Evals are not screenshots

The demo eval is a green check. The year eval is a distribution: how often it is wrong in a way a reviewer will miss. Those are different species. A tool can have a beautiful happy-path rate and still be unusable if the failure mode is “looks right, changes the wrong invariant.”

We tried to build a small eval set from tickets we had already closed. It was humiliating in a useful way. Tasks that looked like the marketing page scored well. Tasks that looked like Tuesday scored like a coin flip with extra confidence. I do not trust a vendor who will not run their agent against your last twenty tickets. I trust one even less who wants to pick the twenty.

Tools look complete until they meet your glue

On stage, the agent calls a tidy set of tools: read file, write file, run test, done. In our stack the work lives in the glue. A deploy is a Jenkins job with a name nobody remembers. The “database” is three Postgres instances and a replica that is lying. The feature flag system is a SaaS that needs a token that is not in the repo. The agent’s tool list does not include any of that unless you build it, and building it is a product, not a checkbox.

I underestimated how much of the year would be spent deciding which tools the agent was allowed to see. Every new integration is a new way to be confidently wrong at a distance. A demo that says “it uses your tools” without showing the failure when the tool returns a 403 is leaving out the common case. 403 is the common case.

The long task is still a fantasy for most of our work

The 2026 pitch is the overnight agent that finishes the epic. I have had a handful of runs that earned the electricity: mechanical migrations, test scaffolding, a dependency bump with a real test net. I have had a longer list of runs that produced a plausible epic and a week of cleanup that I would have skipped if I had just done the first slice myself.

Demos skip the cleanup because cleanup is not cinematic. Cleanup is where the architecture gets quietly rewritten to match the training prior. Cleanup is where the extra crate appears. Cleanup is where the code review becomes the job. If your plan for the year is “agents do the long tasks,” you need a plan for the cleanup that is longer than “we will glance at it.”

People change the system around the demo

This is the omission I care about most, because it does not show up in a latency graph. Once a team believes the demo, they start writing tickets for the agent instead of for each other. Acceptance criteria get thinner. Domain context moves out of the conversation and into a prompt someone will not maintain. Juniors stop asking questions in the channel because the tool might know. Seniors stop writing the paragraph that would have saved the next person, because the next person is assumed to be a model.

I watched a team get faster at producing diffs and slower at producing shared understanding. That is a demo success and an org failure. The keynote will not say “please keep talking to each other.” It sounds like a poster in a break room. It is still the constraint.

The other human omission: someone has to own the misses. In a demo, a miss is a laugh. In production, a miss is a name on a ticket. We had to say out loud that the human who merged the PR owns the outcome, even if the agent typed it. If you cannot say that, you do not have a tool. You have a blame-laundering machine.

None of this means you hide the tools. It means you treat the demo as a lower bound on integration work, not an upper bound on intelligence. The model was never the long pole. The long pole was the ugly door it has to knock on.

What I want from the next demo

Show me the task that failed, and how the product stopped. Show me the permission prompt that fired. Show me the invoice for the looping run. Show me the PR that a tired reviewer should have rejected. Show me the file the agent never saw. If you cannot show those and still want the sale, you are not selling an agent. You are selling a feeling I already had in the auditorium.

I am not anti-agent. I use them. I will keep using them. I just refuse to let a recording that lasted eight minutes set the expectations for a system that has to live for eight years. The year taught me that the missing slides were the job. The green check was the advertisement.