The decisions behind per-turn credential isolation, session history, and race-safe credit enforcement.
The decisions behind per-turn credential isolation, session history, and race-safe credit enforcement.
Building chat over live data sounds straightforward. It isn't.
The moment a CSM types "show me account health for Acme Corp," you have a security problem that a generic AI assistant never has to solve. The AI needs to call real APIs using Acme's credentials. Not your credentials. Not a shared key. Not a demo fixture.
Get that wrong and you have a tenant isolation breach. Here's what getting it right required.
Most chat-over-data tools are single-tenant. You connect your own database, ask questions, get answers. The blast radius of a bug is your own data.
CS teams work differently. A CSM queries on behalf of a customer, with that customer's credentials, scoped to that customer's data. The same AI loop that pulls Acme's invoices might handle Beta Inc's subscription status on the next message. Each call needs to prove it's using the right identity.
Two things need to be true before any API call fires:
Enforcing this at session start is not enough. We enforce it on every turn.
Every time the model wants to call a tool, before credentials load, before any upstream request fires, we verify the combination of organization, connector, and customer account in a single query.
It checks: does this customer account exist, does it belong to the connector being queried, does that connector belong to the authenticated organization, and are both currently active?
If anything is off, such as wrong org, paused account, revoked customer, then the tool call is rejected. The response is deliberately uninformative. The caller gets a 404, not a 403. A 403 leaks information about what exists. A 404 doesn't.
This also means the trust check doubles as the tool catalog load. Resolving which tools are available proves the org/connector/customer triple is valid in the same round-trip. One query, two jobs. There's no way to get the tool list without passing both.
Customer API keys are stored encrypted at rest. When the model needs to call an upstream API, the key is decrypted in-process, injected into the outbound HTTP request, and never returned to the model. The LLM sees the tool result. The data that came back. Nothing else.
This is the same pattern as a good OAuth proxy. The agent acts with the customer's authority without having access to the credential itself. You could log every model input and output without leaking a single API key.
What we do log: which organization, which connector, which tool names were invoked. Not the arguments. Not the credentials.
Session history is a separate thing, and a feature. Conversations are persisted server-side so a CSM can pick up where they left off, share context with a colleague, or revisit a dashboard from a previous session. The distinction matters: we store what the model said, not what it used to get there. Tool arguments, upstream API responses, and credentials are never written to the session log.
Credits meter chat usage. One credit per user message, regardless of how many internal LLM round-trips that message triggers. We had a constraint: no ledger table, no background job, no cron for monthly resets.
The debit is a single atomic SQL UPDATE with a WHERE balance > 0 guard. Two concurrent requests from the same organization both see a balance of 1. Only one UPDATE succeeds. The second gets no row back and surfaces as out-of-credits. No distributed lock, no application-level counter.
Monthly reset is lazy. On each debit attempt, if the period anchor is older than 30 days, we reset first with a WHERE clause that pins on the old anchor value. If two concurrent requests both see an expired anchor and race to reset, only the first UPDATE matches. The second falls through to the standard debit on the freshly-reset balance.
No cron. No clock skew. No "why didn't credits reset this month" tickets.
Each chat turn is a server-sent event stream. We POST a body. The full conversation history on every turn, the browser's native EventSource API doesn't work (GET only). The client opens a streaming fetch and parses frame boundaries manually.
The server runs an LLM tool loop: stream model events, detect tool calls, execute them against the customer's connected APIs, inject results back, repeat. The loop is capped. A stuck model terminates with a clear error instead of streaming indefinitely.
One decision worth explaining: the model can render interactive dashboards directly in chat. Instead of returning a markdown table, it produces a self-contained HTML fragment. A chart, a KPI grid, a comparison view. Rendered in a sandboxed iframe with no external network access. All data is embedded inline. The system prompt tells the model when to use this versus plain prose.
A CSM asking "show me usage by feature for the last 90 days" should get a chart they can paste into a slide deck. Not a wall of JSON.
WHERE balance > 0 is enough to make credit enforcement race-safe. No distributed lock needed.