Every chatbot vendor has a dashboard full of metrics. Total conversations. Average response time. Customer satisfaction. Messages per session. It is an ocean of numbers designed to make the tool look busy. Almost none of it is the metric that matters.
The only metric that matters is containment rate. Specifically: of the conversations the bot started, what percentage ended without the customer needing to talk to a human.
Containment rate is the metric because it is the only one that has a direct line to cost. Every escalation is a human-dollar cost. Every conversation the bot fully resolves is a pennies cost. The containment rate is the exchange rate between those two.
Vendors avoid this metric because it exposes them. A bot that has 10,000 monthly conversations and a 15 percent containment rate is losing you money compared to a static FAQ page. A bot that has 3,000 monthly conversations and 65 percent containment is paying for itself three times over. The raw conversation count is noise. The containment rate is signal.
When we build a conversation bot for a client, we set the target containment rate in the first meeting. Usually 55 percent in the first 90 days, ramping to 70 percent by month 6. If we cannot hit those targets we lose the account, so we do not quote them unless we believe them. That kind of commitment is rare in the vendor landscape, and it is why we can name the number in advance.
The hidden thing about containment rate is that it is mostly a content problem, not a model problem. The model is a commodity. What the model knows is not. Our post on why training data is 90 percent of the work covers this in depth. Containment improvements come from better content, not fancier prompts.
The second hidden thing is that containment rate has to be weighted. A bot that answers "what is your address" correctly 1000 times and misses the refund question 3 times still looks fine on raw containment, but those 3 refunds are where the real money is. Weighted containment by conversation value is the grown-up version of the metric.
If a vendor will not tell you their client containment rates, or will not commit to a number in writing, keep shopping. The good ones can. The rest are selling you a dashboard.