Giving a Support Chatbot Tools, Not Just Documents

Putting a chatbot on your site that answers from your documentation is close to a solved problem now. Upload the pages, paste a snippet, done. The interesting version is the one that can answer "how many people signed up this week" or "set up a waitlist for my new idea", and that turns out to be a different piece of work entirely. Almost none of the difficulty is the model. It is deciding who is asking, what they are allowed to see, and what the bot may do without checking first.

Table of contents

• What a documents-only bot cannot answer
• Identity is the whole problem
• Decide what the model is allowed to see
• Let it act, but never silently
• The failure you will not see
• Test it like software, not by chatting to it
• What this took to build
• Frequently asked questions

What a documents-only bot cannot answer

Feed a bot your help pages and it will handle "what does Pro cost" and "how do custom domains work" perfectly well. Watch real support messages for a week, though, and a large share of them are not about the product in general. They are about this person's account: what did my waitlist do this month, why is my domain not working, can you set this up for me.

A documents-only bot answers those with a paragraph from the manual, which is the polite version of not answering. To do better it needs tools: small, named functions it can call, that read or change something real. The Model Context Protocol is the common way to expose them now, and I wrote about the format itself in what MCP is in plain English.

Worth saying early: the tools are the easy part. Writing one that returns a signup count is an afternoon. Everything below is what takes the rest of the week.

Identity is the whole problem

Your chat provider does not know who your customers are. It knows a visitor by some id its own widget generated in the browser, and it passes that along to your tools. That id is useless to you and, more to the point, anyone can send any value they like.

So the first thing to get right is: your tool endpoint is on the public internet, and it must refuse to act on an account unless the caller can prove it is that account. The shape that works is unglamorous:

• When a signed-in user loads your app, your server signs a short token that says "this is user X, valid until Y", using a secret only your server knows.
• The chat widget carries that token along with the conversation.
• Your tool endpoint verifies the signature and expiry before any tool touches the database. No token, no account access: the bot still answers ordinary questions.

Two details are easy to get wrong. Send the token in the request body or a header, never in a URL, because URLs end up in server logs and referrer headers. And keep the lifetime short, because you are handing a credential to another company's system: ours lasts two hours in the dashboard, and thirty days for a linked Telegram or Slack chat, which cannot refresh it mid-conversation.

Decide what the model is allowed to see

Everything a tool returns leaves your building. It goes to the chat provider, then to whichever model provider answers, and usually into their logs. That makes "what does this tool return" a privacy decision rather than a convenience one.

The rule I settled on: the bot can see counts, survey answers, trends and anything the customer could summarise themselves, and it never sees the email addresses of the people who signed up. Those belong to the customer's own users, who never agreed to anything. When someone asks for the list, the bot points at CSV export in the dashboard instead. Free text gets scrubbed of anything that looks like an email on the way out, because people type addresses into answer boxes.

One more thing that is easy to miss: make "not yours" and "does not exist" return exactly the same answer. If asking about a stranger's waitlist says "not allowed" while a made-up id says "not found", the bot has just become a way to check which ids are real.

Let it act, but never silently

Read-only is a fair place to stop, and plenty of good bots do. If you go further, two rules carry most of the safety.

Preview, then confirm. The create tools return a description of what they would do and change nothing. Only a second call, with an explicit confirmation flag, actually creates anything. The model has to show the user what is about to happen and get a yes in between. It also means a model that misunderstood produces a question rather than a mess.

Enforce your limits again, in the tool. This one caught me out. Our dashboard enforced the free-plan caps by hiding the create button, which is fine for a browser and useless for a bot, since a tool call never sees your interface. Any rule that lives only in the UI does not exist as far as the bot is concerned. Worth auditing your own app for that specifically.

Rate limits matter too, and not only for abuse: a confused model in a retry loop can create twenty things in a minute.

The failure you will not see

Mine broke in the most boring way possible. Everything was deployed, the tools were listed, the token was being signed, and every account question came back with "I cannot see your account". No error anywhere.

The chat platform only forwards that identity token when its engine is new enough to support it, and the engine on the server was one version behind. So it dropped the token and sent the request anyway. Perfectly reasonable behaviour, and invisible: the bot answered, it just answered as an anonymous visitor.

The lesson generalises. Anything in this chain that fails open will fail quietly, and you will debug the model when the problem is plumbing. Test the endpoint directly with a real token, and again with a forged one, before you test it by chatting. And when a tool refuses, make the refusal say which check failed, in words the model will repeat to the user.

Test it like software, not by chatting to it

Chatting to your own bot proves very little. You ask the questions you designed it for, in the words you had in mind. Write the cases down instead, with what each answer must say and must not say, and run them after every prompt change.

The cases worth having are not the happy ones:

• Questions it should refuse, where the only right answer is "I do not have that information".
• Anything with money in it: refunds, discounts, cancellations. A bot that improvises a refund policy is worse than no bot.
• Someone pasting an API key or a card number, to check it is not repeated back.
• Attempts to talk it out of its instructions, including one hidden inside a document in its own knowledge base.
• Your own stale pages. Mine happily quoted a blog post that still advertised a free plan we removed months ago, which is its own kind of lesson.

Some of it does not need a model to grade. A plain search for anything shaped like an API key or a card number in the reply is cheaper and more reliable than asking another model whether the answer leaked something.

What this took to build

For the chat side I used chatfrom.io, which handles the widget, the knowledge base and the conversation, and calls out to tool servers you run yourself. That split is the part that matters: the chat product never sees the database, and the tools never see the conversation. It also passes the signed token through to the tool servers you nominate and nowhere else, which is what made the identity design above possible without inventing a protocol.

On the Lighthouse side it came to one public endpoint with a handful of tools, a token signer, and rather more tests than tools. Two kinds of tool, kept firmly apart: public ones anyone can call, such as live pricing, and account ones that do nothing at all without a verified token. The same endpoint also answers our own customers' AI clients, which I wrote about in connecting your waitlist to Claude with MCP.

Unglamorous, but the ratio is the point: a few hundred lines of tools, and everything else spent on who is asking and what they may do.

Frequently asked questions

Is a support bot worth it before you have many customers?

For answering support, usually not: with a handful of customers you should be answering personally, for the reasons in answering support when you are the whole company. For letting people do things in your product by asking, it is a product feature rather than a support saving, and that is a different decision.

Can I just give the bot my database credentials?

No. Every tool should be a narrow function with the account baked in, not a way to run queries. The whole security model here is that the bot can only do the specific things you wrote, for the one account that proved who it is.

What happens when the token expires?

The account tools stop working and the bot says so. In a browser that is invisible, because a page reload issues a new one. In a linked Telegram or Slack chat the person has to link again, which is the main reason to pick a longer lifetime there.

Does the model see the token?

It should not. It travels as a header alongside the tool call, not as part of the conversation, so it never lands in the transcript or in a trace. Worth checking on whichever platform you use, because "opaque user id" fields have a habit of appearing in logs.

How do you stop it inventing things about your product?

Give it a tool for the facts that change, and tell it to prefer the tool over anything in its knowledge base. Pricing is the obvious one: ours reads from the same file the pricing page uses, so the two cannot drift.

If you are thinking about this for your own product, the order that saved me time: start read-only, get identity right before anything else, decide deliberately what may leave your servers, and only then let it create things, behind a confirmation. The model will be the least surprising part of the whole exercise.

Lighthouse is a waitlist, survey, newsletter, and feedback toolkit for indie founders, with an MCP server so your own AI client can query your signups and feedback. From an indie dev, for indie devs and makers.