A year of shipping with AI coding agents: what actually stuck

I led platform teams through two AI tool rollouts before this last year, and I still underestimated how quickly a demo becomes a habit you have to unlearn. We did not “adopt agents.” We shipped production software with them in the loop, every week, on services that already had customers. Some of the workflow survived. Most of the theater did not.

This is the residue. Not a vendor scorecard. Not a prediction. The practices I would keep if every agent product I used this year disappeared tomorrow and I had to reconstruct the useful parts with whatever came next.

What I thought would stick, and didn’t

I thought the win would be speed-to-first-draft on greenfield services. Sometimes it was. More often the first draft was a plausible architecture for a system we were not building, with a test suite that asserted the happy path and a README that sounded like a launch post. The time we saved on typing we spent on deleting confidence.

I thought chat-driven refactors of large modules would replace the afternoon I used to block off for them. They replaced the first hour. They did not replace the hour where you discover the module had three callers that were not in the prompt. Agents are local thinkers unless you force them to be historians. We did not, at first. We paid for that in regressions that looked like they had been reviewed.

I thought “the agent writes the PR, the human reviews” would be a stable division of labor. It is a stable slogan. In practice, a PR written by something that does not feel embarrassment needs a different review than a PR written by a person who will get paged. We had to change the review, not just the author line.

What actually stuck

Narrow tasks with a done definition

The agents that earned a place in the week were the ones I pointed at a task I could describe in one sentence plus a test. “Add the timeout to this client and make the existing integration test fail if it hangs.” “Generate the migration and the rollback, do not touch the app layer.” “Explain why this flaky test died on CI and only CI.”

The ones that did not stick were “improve this service” and “help me think about the design.” Those are conversations. I already have colleagues for conversations. An agent in that mode produces a slide deck in prose. I can write a slide deck.

Stuck practice: I write the acceptance check first, even if it is ugly. If I cannot write the check, I do not open the agent. I open a document. That sounds puritan. It saved us from a month of generating architecture that nobody could merge.

The repo as the prompt

Context windows got larger. That did not make dumping the monorepo into the prompt a strategy. What stuck was the opposite: a small, current map. A CONTRIBUTING note that says how we run tests. An AGENTS or equivalent file that names the packages you must not “clean up.” A test command that works. The teams that treated the agent like a new hire with no onboarding got new-hire output: enthusiastic, slightly wrong, proud of deleting the thing that looked unused.

I started keeping a living “do not” list next to the build instructions. Do not upgrade the framework. Do not reformat the file. Do not invent a utility package. Do not add a dependency whose name you saw in a tutorial. The list is rude. Rude lists survive contact with autocomplete.

Diffs I can hold in my head

We had a stretch where the agent would return a 40-file change that was, in some abstract sense, correct. Nobody on the team could review 40 files before lunch. Those PRs either sat or they merged with a shrug. Both outcomes are how you get a production incident with a polite commit message.

What stuck: I throw away work that I cannot review in one sitting. I ask for the smallest commit that makes the check green. I would rather run the agent four times than accept a novella. This is not a productivity loss. This is how we still know what the system does.

I also stopped letting the agent write the PR description as marketing. The description has to say what might break. If the agent cannot name a risk, I name one before I request review. “No risk” is a smell. Everything that touches money, auth, or a queue has a risk.

Tests as the handshake

I already believed in tests. I did not understand how much of the agent workflow is just a worse version of TDD until we leaned into it. The handshake that stuck is: human writes or sketches the test, agent implements, human reads the diff against the test, human runs the parts the agent did not run.

Agents will write tests that encode the bug. They will also write tests that mock so much of the world that the test is a compliment. The review I now do first is not style. It is “what would still pass if the implementation were a no-op or a lie.” If the answer is “most of this file,” we delete the file and start with one ugly integration check.

On-call stayed human

We tried letting an agent draft the incident doc and the first patch. The draft was useful as a timeline once we had already understood the incident. It was dangerous as a first responder. It would confidently blame the last deploy because the last deploy is a good story. Twice it was wrong. Once it was right in a way that made us stop looking at the cache, which was the actual problem.

What stuck: agents summarize after a human has a theory. They do not get the pager. They do not get to close the incident. I will die on this hill and I have the two postmortems to sit on it with me.

The social stuff that surprised me

Juniors did not become obsolete. Juniors who only submitted agent output became invisible. The juniors who stayed visible were the ones who could say “I asked for X, it did Y, I kept Z, I rejected the rest because of this caller.” That sentence is the job now. We started asking for it in standup the way we used to ask “what did you work on.”

Seniors did not become ten-times engineers. Seniors became people who spent more of the week reviewing and less of the week typing, which is a trade I will take, and also a trade that burns you out if you do not put a cap on review load. We had to schedule “no agent PRs after 4” the way we used to schedule no deploys on Friday. Review is work. A flood of plausible diffs is a denial-of-service attack on the people who still understand the system.

I stopped counting lines generated. It is a vanity metric with a vendor behind it. We count escaped defects, time-to-green on the tasks we chose, and how often a change had to be reverted. Those numbers got better on mechanical work and worse on the weeks we used agents as architects. That split is the year, in one sentence.

The stack that earned a place

I am not going to name a winner. The products changed names twice while we were using them. The pattern that stuck is more stable than the logo.

Inline completion for the line I was already going to type. I would miss it tomorrow. It is not an agent. It is a better keyboard.

A repo-aware agent for the bounded task, run locally, with the test command in the prompt and the “do not” list in the repo. I would miss it on migrations, boilerplate, test scaffolding, and “why is this type exploding.”

A second pass that is not the same agent, or is the same agent with a hostile prompt: “what did we just break.” I would miss it the way I miss a grumpy reviewer. I do not let it merge.

I would not miss autonomous overnight runs that “just finish the ticket.” We tried. We got a ticket that was finished in the way a hotel room is cleaned if you stuff everything in the closet. The closet was our dependency graph.

What I changed in my own week

I write more specifications that a machine can fail. Not novels. A paragraph and a command. I protect two hours a day that are not review and not prompting — the hours I still spend in the code with my own hands, because I refuse to become a person who can only manage a system I no longer know. That sounds romantic. It is maintenance of the only skill that makes the review possible.

I keep a private log of agent misses that almost shipped. Not to be precious. To keep the fear calibrated. After a month of good luck you start merging with your shoulders down. The log is a way to put the shoulders back up without becoming a Luddite.

And I tell product the truth about the speedup. Mechanical work is faster. Design work is not. If they heard a keynote that said otherwise, they heard a keynote. Our burn-down on well-specified tickets improved. Our calendar of “we need to think” did not shrink. Anyone who sells you a shrink on that calendar is selling you a future incident.

If I had to start the year over

I would ban repo-wide refactors from agents in the first month. I would require a test before the first prompt on anything that touches money or auth. I would add the “do not” list before I added the tool. I would cap the size of a generated PR in the team agreement, in writing, like a lint rule. I would not hold a lunch-and-learn about prompts. I would hold one about review, because review is the new bottleneck and we pretended it was still typing.

What stuck was not intelligence. What stuck was constraint: small tasks, written checks, diffs a human can hate in an hour, a pager that still rings for a person. The rest was a very expensive way to generate text that looked like work.

I am not going back to a world without the keyboard that finishes my line or the agent that drafts the migration. I am also not going back to the month we let the demo set the workflow. The demo was always going to be the demo. The year was for finding the parts that survived a real service.