Why AI still doesn’t ship without a real engineer in the loop

I have let a model write a feature end to end. Tests, types, the commit message. It looked like shipping. It was not shipping. Shipping is the hour after merge when the webhook fires twice, the flag is wrong for one tenant, and someone has to decide whether to revert. The model will draft the revert. It will not own the customer who is waiting, and it will not sit in the incident channel with its name on the rotation. That is still a person. I am that person more often than I am the person who typed the first draft.

This is not a eulogy for typing. Typing got cheaper. Shipping did not. Here is why I still will not take the engineer out of the loop, even when the loop is mostly review.

The loop is not “write code”

The loop is: decide what is allowed to change, make the change, prove it in the environments we actually have, expose it to a fraction of users, watch the metrics that are not the demo, and be reachable when the proof was a lie. Models are strong at the middle of that sentence. They are weak at the ends. The ends are where money and trust live.

I have watched a generated migration apply cleanly in staging and lock a table in production because staging was a toy. An engineer who has been burned once will ask about table size. A model will ask if you want indexes. Both questions are useful. Only one of them comes from having been paged.

I have also watched a model invent a feature flag name that did not exist and a rollback plan that assumed a tool we do not run. Fluent. Wrong. The loop needs someone who can say “we do not have that” without needing a prompt.

Someone has to choose the blast radius

Every change has a radius. Models will optimize for the ticket as written. Tickets lie by omission. “Add a field” can mean a new column, a backfill, an API version, a client that is still on the old app store build. An engineer in the loop is the person who names the radius before the PR is polite. I have skipped that and merged a “small” generated change that was a mobile-breaking rename. The model did what I asked. I asked badly. The job is to notice I asked badly before users do.

Flags, batches, dual writes: those are loop skills. I will use a model to write the dual-write. I will not let it decide that we do not need one.

Someone has to be wrong in public

Incidents need a name. Not a bot that summarizes Slack. A name that can say “this is on me, here is the next check.” Models can draft the timeline. They cannot take the social hit or the legal one. They cannot call the customer. They cannot decide to notify early versus wait for a clean story. Those are judgment calls with a face.

I have used a model to write the first incident doc. I still had to delete the paragraph that sounded sure about a cause we had not proved. Sure is a style the models love. Sure is how you ship a wrong RCA and train the team to stop looking.

Someone has to refuse a good-looking change

The dangerous PRs now look finished. Types pass. Tests pass because the tests were generated to match the code, not the world. An engineer in the loop is the person who asks what the test does not say. I have started requiring one test the model did not write, or one I rewrote, on anything that touches money or auth. Not as ritual. As a check that a human still knows the sad path.

I also refuse changes that expand the dependency surface for a convenience. Models love adding a package. Packages are a supply-chain and an on-call. That refusal is not in the ticket. It is in the scar.

Someone has to hold the product, not the snippet

A model will complete the function you highlighted. It will not remember that this function sits behind a legal sentence, or that we promised a customer we would not store that field, or that marketing already announced a date. Those constraints live in people’s heads and in threads the repo does not index well. An engineer in the loop is a retrieval system for the company, not just for code.

When the index is good, the model gets closer. When the company is a mess of side channels, the model is confidently local. Shipping is rarely local. Shipping is the join of code, policy, and a calendar. I will not take the join out of a human’s hands because the function looks pretty.

What I did take out of the loop

Boilerplate. First drafts of tests I will edit. Commit message spelling. The third time I write the same mapper. The “what files might be related” search. Those are real hours back. I am not romantic about them. I am romantic about the merge and the page. Those stay human.

I also took out the fantasy that a junior ticket queue is a training ground that will survive unchanged. The queue got faster. The training now has to be the loop: radius, refusal, incident, product join. If we only train people to accept the model’s PR, we are training reviewers who cannot refuse. That is how you get a team that ships and cannot explain.

What happens if we pretend the loop is empty

I have seen the experiment in small: merge what the agent opened if CI is green. CI was green. A permissions check used a helper the model copied from a test. Production users were not the test user. We spent an afternoon on a hole that looked like a style win in review. The empty loop is not faster. It moves the time to after users find it. That time is more expensive and less kind.

I have also seen the opposite failure: so much fear of the model that we retype everything and call it craft. That is not a loop either. That is denial of a tool that already writes the first draft. The engineer in the loop is not a typist. They are a gate with context. If we make them a typist again, we will lose them, and then we will have only the empty loop.

Regulated work makes this less abstract. Someone signs. Someone’s name is on the change record. I will not put a model’s session id on that line. I will put the engineer who said the radius was acceptable. If a company wants to pretend otherwise, they can talk to their counsel. I will not help them confuse a completion with a signature.

The loop I will defend in a staffing review

One named owner per change that can reach production. They may have used a model for 80% of the diff. They still say yes. They are still on the rotation for that surface. If we cannot name the owner, we do not merge. “The agent opened the PR” is not a name.

A review that is allowed to be slower than the generation. Generation is cheap. Review is the product. I will not measure engineers by how fast they accept. I will measure whether the sad path was considered and whether the radius was named.

A kill switch for anything the model touched that talks to a user or a ledger. I learned that on other stacks. I apply it here. Fast drafts make it more necessary, not less.

AI still does not ship without a real engineer in the loop because shipping is not a completion. It is a decision with a blast radius, a face in the incident, a refusal of a pretty diff, and a memory of the company the repo cannot hold. I will keep using the models. I will not let them be the name on the merge. The name is the job. The job got more about the name, not less. That is the part the demos keep skipping, and it is the part I still get paid to do on a Thursday night when the webhook fires twice. If a founder asks whether we can staff fewer engineers now, I will say we can staff fewer typists. I will not say we can staff fewer names. The names are how software stays a product instead of a pile of green builds. I will keep paying for names. I will let the models type.