What the AI Should Not Decide

A conversational assistant can answer a buyer at 11 pm in under 10 seconds. It can also tell a nervous first-time borrower something that sounds like a rate quote. The speed is the gift and the risk. Here is where I draw the line between what an AI layer handles and what a person has to keep.

A lone figure stands in a far white doorway at the end of a long stone arcade while a man in the foreground watches
The Good, the Bad and the Ugly (1966), dir. Sergio Leone

A conversational assistant can answer a buyer at 11 pm on a Sunday in under 10 seconds.

That is the gift. The person has been looking at listings for an hour, they have a question about whether they could afford one, and instead of waiting until Monday to hear from a human, they get a response before they close the tab.

It can also tell a nervous first-time borrower something that sounds exactly like a rate quote. It is polite and fluent, and the person walks away believing they were promised something. That is the risk, and it sits right next to the gift.

I want to be useful about where the line sits, because both extremes are mistakes. An AI layer that does nothing wastes the most valuable minutes in the funnel. An AI layer that decides things it should not creates problems nobody wants to explain to a regulator or a client.

Speed to lead is an AI-shaped problem

I have written before about the first five minutes after a form submission. The research I have seen points the same way: a fast first reply tends to win, and the odds of a conversation fall the longer someone waits. A person who submits an inquiry at 9 pm is not going to be reached by a human until the next morning in most shops.

This is where language models earn their keep. They are fast. They do not sleep. They can be patient with a person who asks the same question three ways. They can take a messy, half-formed question and turn it into something a loan officer can act on without a second round of “so what are you looking for?” That is a real contribution, and I think the shops that lean into it well will win ground over the ones that do not.

I do not want to oversell it. But the case is straightforward: the model is good at being available, and availability is exactly what the first minutes require.

What the AI can decide

There is a useful set of things the model can handle by itself, and I will stay at the category level.

It can answer general questions about the process: what documents are usually needed, how long a typical closing takes, what the difference is between two loan types in plain language. It can gather context, such as the property type, the timeline, and what the person is trying to do. It can schedule a call and send a confirmation. It can notice urgency, like a buyer with a contract deadline in 2 weeks, and flag that to a human right away. It can sort a conversation into a clean summary.

None of that is a commitment. None of it requires the model to know anything the person’s file would reveal, and none of it should put words in the company’s mouth that it cannot stand behind. That only holds if the answers come from approved facts, because a model will happily invent a confident policy if you let it.

What it must not decide

This is the more important list.

Rates. A quoted rate is a specific, regulated statement that depends on facts the model does not have, and it should never invent or guess one. Approval. Whether someone qualifies is a decision with legal weight, and no fluent paragraph substitutes for it. Eligibility, programs, and anything that sounds like advice about what a person should do with their money. Anything that reads as a promise.

The issue is not that the model is dumb. It is that it is persuasive. A frightened person at 11 pm will take a confident sentence as a fact. I wrote recently about how a fluent sentence is not a compliant one, and this is the same problem on the lead-handling side. The model’s tone gives its words more authority than the model has earned.

The design response is to build the line into the system, not into the prompt alone. The layer should have a defined set of things it will not say, and a defined way to hand off when the conversation moves toward them. I will not describe the specific rules or how they are enforced. They are part of how I build for clients. But I will say the handoff is not optional, and it is not decoration.

The handoff is the product

If I had to name the single piece that decides whether an AI layer helps or hurts, it would be the handoff.

Imagine the good version. A buyer chats at 11 pm. By 9 the next morning, a loan officer opens the conversation and sees a summary: the person’s goals, their timeline, the question they were nervous about, and the two things the assistant deliberately did not answer. The loan officer calls. The buyer does not have to repeat anything. The first human interaction starts warm.

Now the bad version. The assistant answers a few questions, then drops the person into a generic “someone will contact you.” The loan officer gets a name and a phone number. The buyer, who spent 20 minutes explaining their situation, gets asked to explain it again. All the speed was spent, and the effect was to make the experience feel cheaper than a plain phone call.

A bad handoff cancels the benefit of the whole layer. I would take a slower assistant with a clean handoff over a faster one that dumps context on the floor.

Say that it is an AI

There is a trust question that I think is simple. If a person is talking to an assistant at 11 pm, they should be told in the first message. A lot of people will happily use one and be glad it exists. Fewer will be happy to find out later that they were talking to software while believing it was a person.

It should also be easy to ask for a human, and the system should honor that quickly. And the assistant should be comfortable saying “I am not the right one to answer that,” and meaning it, rather than improvising an answer to avoid disappointing someone.

I will admit the first version of any of these is usually too eager. It answers things it should have passed along, because answering feels helpful. The tuning is mostly pulling it back toward humility, and it is never really finished. I would rather say that than pretend an assistant can be configured once and left alone.

Where I land

Let the model handle the first minutes and the sorting. Keep every promise with a person.

The goal is a faster path to a real conversation, not a replacement for it. If the AI layer makes the human conversation better prepared and better timed, it has done its job. If it starts making decisions that belong to a licensed professional, it has gone too far, and no amount of speed makes up for that.