I shipped Spring AI in production. Here’s what I’d do differently next time

The first production Spring AI feature I shipped was a “summarize this ticket” button. It looked harmless. It was a ChatClient, a prompt, a stream into the UI. Support liked it for a week. Then we had a wrong-tenant sentence, a bill that did not match the traffic we thought we had, and a Friday where the model vendor hiccuped and the ticket page hung. I did not pull the library. I changed how I will ever ship the next one.

What the framework is for is already in what Spring AI actually buys you. This is the after-action: the list I keep next to the pull request template now.

I would ship the kill switch before the prompt

We put the feature flag in late, after the demo had already trained people to expect the button. When I needed to turn it off, I had to race a deploy. Next time the flag exists in the first commit: off by default, per tenant, with a metric on how often we flip it. A model outage is not a reason to take down the ticket page. The page should render. The summary should say it is unavailable. That is a product decision I should have written before I wrote the system prompt.

I would also put a budget on the bean. Max tokens in, max tokens out, max calls per ticket per hour. Spring AI will happily retry and stream. I would not let it. The first bill conversation was me explaining that “it is just a summary” still counts when every agent clicks twice.

I would not put the model on the GET

We generated on page load because it felt magical. It was also a completion on every refresh, every support train, every deep link from Slack. Next time the summary is an explicit action or a job. Cache the result against a content hash of the ticket. If the ticket did not change, do not pay again. If the hash changed, the old summary is stale and we say so. Silent stale summaries are how you get someone acting on a paragraph from Tuesday.

Streaming on GET also made the page a hostage. When the vendor slowed down, the spinner became the product. Next time the UI paints first. The model fills a panel. If the panel fails, the ticket is still there. I had to learn that the hard way in front of a queue of agents.

I would write ten cases before I write the advisor chain

We launched on vibes. “It looks good on these three tickets.” Then a fourth ticket used a customer’s internal nickname and the model invented a status we do not have. Next time I want a folder: input, retrieved chunks if any, properties I will assert — must mention the ticket id, must not mention another org, must not invent a state. I re-run that folder when I change the model, the prompt, or the retriever. Spring’s tests will inject a stub. The folder is for the live model in a staging project. Stubs do not catch the invention.

I would not let an advisor stack grow until I cannot print the final prompt. Next time I log a redacted prompt id and a hash, not the full text, in the default APM. Full text lives in a store we control, with a retention we can defend. Legal asked for a sample. The sample had a phone number. That was an afternoon I will not repeat.

I would treat retrieval like a query with a tenant id

The wrong-tenant sentence was not a model being evil. It was a vector query that did not inherit the ACL we already had in SQL. Next time the retriever takes the same authz the ticket GET takes. If I cannot filter by org in the store, I do not put other orgs’ documents in the store. Separate indexes are boring and I will take boring. I would rather a miss than a leak.

I would also show the agent the passages we used, or at least a “based on these two comments” line. When we hid the sources, the summary became scripture. When we showed them, people argued with the right object. That is the product I meant to ship.

I would put timeouts on the client the first day

We inherited generous defaults and then wondered why a thread pool filled up. Next time: connect timeout, read timeout, a circuit that fails open to “summary unavailable.” I would not retry a 30-second hang three times on a user click. I would retry a job. User click gets one try and a button to try again. Jobs get a queue and a dead letter I will look at.

Tool calling made this sharper on the second feature. A tool that hits our own API can deadlock if that API waits on the model. Next time tools are internal, cheap, and not on the same request path as the completion that called them. I drew that box after an incident, which is the wrong order.

I would not swap models because a slide said cheaper

We flipped a property to a smaller model to save money. The summarizer was “fine.” The tool-calling flow stopped calling tools. The framework made the flip look safe. The behavior was not portable. Next time each use case pins a model and a prompt version. Cheaper is a project with the eval folder, not a Friday config change.

I would also stop treating “we can switch vendors” as a reason to skip a contract. Portability is a buy for the client shape. It is not a buy for the product. The product still has to pass the folder.

I would keep a human on anything that leaves the building

The ticket summary stayed on the page. The next request was “draft the customer email.” I delayed that until there was an edit box and a send that was still the agent’s. Next time I will not let a completion go to a customer without a human click. Internally, I still want a human on anything that changes money, access, or a legal sentence. The model can draft. The bean does not get send().

I would staff the on-call like it was a dependency

We added a model and left the runbook as “restart the pod.” The failures were not pod failures. They were empty tool calls, truncated streams, a 429 from the vendor, a sudden style change after a silent model update. Next time the runbook has those four, with the flag name and the person who owns the prompt. If nobody owns the prompt, we do not ship. I do not want a library without an owner any more than I want a database without one.

I would also tell support what the feature is allowed to be wrong about. “May miss a comment” is different from “may invent a status.” We did not write that, so every miss felt like a defect of the same size. Agents need a severity, not a vibe that the AI is moody.

I would version the prompt in git, not in a wiki

The first prompt lived in a doc someone edited without a review. The second lived in config. Both drifted. Next time the prompt is a file next to the code, reviewed like code, tagged with the eval folder version. If product wants a tone change, that is a pull request. If they want it today without the folder, they can have a meeting instead of a deploy. I lost an argument about this and then I lost a week to a tone change that also dropped a safety sentence. I will not lose that argument quietly again.

What I would repeat

Spring AI as the client, not as the product. Metrics next to the other HTTP clients. A ChatClient I can mock. Structured output for extract-and-fill, not for “be helpful.” A team that already lives in Boot, so the config is not a second career.

I would repeat the narrow feature. Summarize a record the user can already see. Extract fields into a form. Classify for a queue with a human override. I would not repeat “assistant for the whole app” as a first ship. That is how you get a second product inside the first one, with no evals and a Slack channel full of almost.

Next time I will still use Spring AI in a Spring shop. I will not start with the prompt. I will start with the flag, the budget, the timeout, the tenant filter, and the ten cases. The prompt is the easy part. I treated it like the only part. That is the thing I would undo first. If a founder asks how long “adding Spring AI” takes, I will give them two numbers: the client, and the product around the client. I will not let those numbers get averaged into a sprint named AI. Averaging is how we shipped a button and then spent a quarter paying for it.