LLMs in real Spring projects: what Spring AI actually buys you — and what it doesn’t

The first time a PM asked me to “just add Spring AI,” they said it like it was a starter. Flip a dependency, get a chatbot, ship in a sprint. I have now put Spring AI into two production services and pulled it out of a third. It is a real library. It is not a product. What it buys you is a Spring-shaped way to talk to models. What it does not buy you is a reason to talk to a model, or a way to know the answers are any good.

This is the buy list I use in a design review. Not a tutorial. Not a victory lap. If you already shipped it and want the scars, that is a different article. This one is what the framework is actually for.

What Spring AI actually is

Spring AI is an abstraction over chat models, embeddings, vector stores, advisors, and tool calling, wired the way Spring people already wire datasources and rest clients. You get a ChatClient, configuration, observability hooks, and a portable-enough API that switching from one hosted model to another is a properties change instead of a rewrite of every call site. If you live in Boot, that is not nothing. It is the same gift JDBC templates were: you still write SQL, you just stop inventing a client every time.

It is not a brain. It is not a RAG product. It is not an eval harness. It will not stop you from putting the model on the request path of a checkout. It will make that mistake look like idiomatic Spring.

What it buys you: a place for the call to live

In a real Spring app the win is plumbing. Retry, timeouts, API keys in the environment you already use, metrics next to the rest of the HTTP client metrics, a bean you can mock in a @SpringBootTest. I have written the “just use the OpenAI SDK in a @Service” version. It works until you want a second provider, a second use case, and a trace that does not look like a one-off. Spring AI is the one-off becoming a citizen.

It also buys you structured output and tool calling in a shape your team can review. A record for the model to fill, a function the model can invoke that is still your Java. That is useful when the feature is “extract these fields” or “look up this order and then draft.” It is not useful when the feature is “be generally helpful.” Generally helpful is a demo.

Portability is a partial buy. You can point the same ChatClient at a different model. You cannot assume the prompts and tools still behave. I have watched a swap “just work” on a summarizer and fall over on a tool-calling flow because the new model was polite and never called the tool. The framework made the swap cheap. The behavior change was still ours.

What it buys you: RAG pieces, not a retrieval design

Spring AI will talk to a vector store. It will embed chunks. It will stuff context into a prompt. That is the easy 20%. The hard 80% is what you chunk, how you update it, what you do when the store is stale, how you keep customer A’s tickets out of customer B’s prompt, and how you know the retrieved passage was the right one.

I have seen teams celebrate “we have RAG” because a vector store bean started. Then support copied a paragraph from the wrong tenant’s FAQ. The framework did not fail. The tenancy model did. Spring AI will not invent your document ACL. If your app already has a hard time getting the right row out of Postgres, a vector index will not make you honest.

Advisors are a buy I like when they stay small: logging, a simple memory window, a retrieval step you can read. Advisors are a smell when they become an invisible pipeline nobody can debug. If I cannot draw the prompt as it left the JVM, I do not want another advisor.

What it does not buy: evals, cost, or a use case

Spring AI will not tell you if the feature is any good. You still need a set of cases you re-run when you change the model, the prompt, or the retriever. I have shipped without that and then spent a month arguing about “it feels worse.” Feelings are not a release gate. A folder of inputs and expected properties is. The framework does not give you the folder.

It will not cap your bill. Token counts are your problem. Caching is your problem. “The model is in the request path so every page view is a completion” is your problem. I have watched a helpful sidebar turn into a finance conversation because we streamed a summary on every load. Spring made the call easy. Easy is how the bill grows.

It will not turn a bad product idea into a good one. If the user needed a filter and a date picker, a chat box is a slower filter. If the user needed a deterministic calculation, a model is a liability. The library is happy either way. Your incident review will not be.

What it does not buy: safety, tenancy, or “the model is just another bean”

Prompt injection is still your problem. Tool calling makes it sharper: the model can now hit your functions. I treat every tool like an HTTP endpoint with a hostile caller. Authz stays in your code. The model does not get a session it did not earn. Spring AI will invoke the function you registered. It will not ask whether that function should exist for this user.

PII in prompts is still your problem. Logging the completion is still your problem. If your observability advisor dumps full prompts into a third-party APM, you have a data story, not a Spring story. I have had to turn that off after legal read a sample. The default was convenient. Convenient is not a DPIA.

People say “it is just another bean.” It is not. Beans you wrote fail in ways you can replay. Models fail in ways that look like success: 200, fluent, wrong. Your tests need to pin behavior you care about, not assert that a string came back. Spring’s test slice will help you inject a stub. It will not invent the stub’s contract. You will.

When I would reach for it

A Spring shop. A narrow feature: extract, classify, draft with a human in the loop, summarize a record the user already has the right to see. A need to keep the call next to the rest of the platform’s metrics and config. A team that will own prompts like they own SQL.

I would not reach for it to build a general assistant for the whole product. I would not reach for it if the team has never called a model before and thinks the dependency is the skill. I would not reach for it to replace a rules engine that already works. I would not reach for it in a service whose on-call does not read Java, just because the rest of the company is Spring. Ownership still beats fashion.

When I would skip it and call the model myself

If the feature is one endpoint and one vendor and we will not switch, a thin client can be less magic. If we already have an internal LLM gateway, I would talk to the gateway, not add a second abstraction. If the team is not a Spring team, I would not add Spring to get Spring AI. That sounds obvious. I have seen the opposite proposed so the architecture diagram would say “standard.”

If we need streaming with a weird protocol, or a model the starters do not treat well, I would rather a small adapter we own than a fight with the abstraction. Abstractions are a buy until they are a second bug tracker.

The sentence I put in the design doc

Spring AI buys us a consistent way to configure, call, observe, and test model access inside an app we already know how to run. It does not buy us retrieval quality, evals, cost control, tenancy, or a reason for the feature to exist. If those lines are empty, I do not add the dependency. If those lines are written, I am happy to let the ChatClient be as boring as a JdbcTemplate. Boring is the compliment. The compliment is not “we have AI now.” The compliment is “the call lives in the platform, and the hard parts still have names.”

I still get asked to estimate “the Spring AI work” as if it were a module. I estimate the feature: the cases, the tenancy, the kill switch, the bill. The module is a day or three. The feature is the quarter. Teams that invert those numbers ship a bean and then live in the slack channel where the answers are almost right. I would rather ship the kill switch first. Spring AI makes the call easy enough that you need the switch more, not less.