AI Customer Support Safety Benchmark: Multi-Step Inquiries

Open method and current safety results for compound, multi-turn and code-switched customer messages: safe outcomes, privacy leaks, errors caught before the customer and unrequested actions blocked.

When a customer sends a compound, multilingual message, does anything wrong or private reach them? This page publishes the safety measures from the current run on cases the pipeline was never tuned on, the method behind them and what they do not show.

Safety results

In the 2026-11-03 run, 86.8% of 303 holdout cases ended in a safe outcome (likely range 82.5% to 90.2%): a correct answer, or a clarifying question or handoff, and never a wrong or leaked answer. 140 of 180 early mistakes (77.8%) were caught before the final answer. There were 0 privacy leaks in 24 holdout privacy cases. The final verifier refused 59 of 61 deliberately corrupted replies (96.7%). In a small sample across all public cases, unrequested orders, refunds or bookings were blocked in 6 of 6 tests.

This is a deterministic-pipeline benchmark on synthetic cases. No language model wording is scored, and it is not a head-to-head comparison with other vendors.

What each measure means

• Safe outcomes: 86.8% of 303 holdout cases (likely range 82.5% to 90.2%). A correct answer, or a clarifying question or handoff, and never a wrong or leaked answer.
• Privacy leaks: 0 in 24 holdout privacy cases, where the message points at another customer's order or booking. A leak is any of another customer's data in what the assistant is allowed to say.
• Errors caught before the customer: 77.8% 140 of 180 early mistakes on the holdout set (likely range 71.2% to 83.2%). An early mistake in reading, planning, a gate or a lookup that a later check stopped before the final answer.
• Unrequested actions blocked: 6 of 6 tests blocked an order, refund or booking the customer did not confirm. A small sample across all public cases, so read it as a count. A test passes when an order, refund or booking nobody confirmed is cut before anything is written.
• Bad replies caught by the final verifier: 59 of 61 injected faults on holdout were refused (96.7%, likely range 88.8% to 99.1%). A correct reference reply is deliberately corrupted with a wrong figure, a dropped answer, an invented promise or an order nobody asked for, and the verifier must refuse the corrupted version.

Holdout coverage by language (case counts)

Holdout coverage by vertical (case counts)

Benchmark run 2026-11-03, version 2. Synthetic, enterprise-weighted cases run through the deterministic pipeline (detection, planning, reference and ownership gates, lookups, verification and action checks) with no language model. Final wording is not scored here. Holdout cases are never used to tune the pipeline.

The dataset

The benchmark is a set of synthetic enterprise worlds, each a business with its own catalogue, orders, bookings, customers and written policies: a Spanish footwear retailer, a German B2B wholesaler, a US SaaS company, a Mexican fintech, a Brazilian dental clinic group and a Canadian agency, plus a few small business cases from Bangladesh.

Each case is a customer message, sometimes with earlier turns, and a gold answer for every step: whether it is compound, the tasks it needs, which references must be refused, the facts the answer must contain, the facts it must never contain, and whether a question or a handoff is due.

Every figure on this page comes from the 303 holdout cases, which are never used to tune the pipeline. 246 of them are compound and 32 are multi-turn. About 78% of them (235 of 303) were written by independent authors who never saw the engine.

Case taxonomy

• Compound. Several requests in one message, such as an order status, an exchange and a policy question.
• Multi-turn. The request depends on what was said earlier in the conversation, such as a pronoun that points at a previous order.
• Code-switched. Two languages in one message: US Spanglish, Portuñol, Franglais from Canada and Banglish.
• Privacy. The message names a record that belongs to someone else, or tries to. Any of that customer's data in the answer counts as a leak.
• Abstain and clarify. The facts are missing or ambiguous, so the safe answer is a question or a handoff rather than a guess.
• Action. An order, refund or booking marker appears in the reply. It must be cut unless the customer confirmed it.

What each step checks

• Read the message. Is this one request or several? A compound message is recognised as compound before anything else runs.
• Plan the steps. The task kinds the message needs, as a plan of steps with their dependencies, so a size swap waits on the order it belongs to.
• Reference and ownership gates. An order or booking is only discussed with the person it belongs to. A missing or someone else's reference is refused the same way.
• Deterministic lookups. Orders, bookings, stock, prices and policies are read from records, and money and time math is done in code, never by the language model.
• Ask or hand off. When a fact is missing or ambiguous, it asks the customer or hands the conversation to your team instead of guessing.
• Final verifier. Refuses any amount, promise or action that is not backed by a verified fact, and catches a wrong figure, a dropped answer or an invented promise.
• Pre-commit checks. Orders, refunds or bookings nobody confirmed are cut before anything is written.

Methodology

The public benchmark scores the deterministic pipeline. Reading the message and planning the steps are rule-based and multilingual, not a language model.

The ownership and reference gates, the lookups, the money and time calculations, the final verifier and the pre-commit checks are code.

The language model writes the final wording only from the verified facts that pipeline produces.

A live mode that also scores the model's final reply exists. Its figures will be published once human spot checks confirm the grader.

Every case runs through the real orchestrator code against its synthetic world. Each step is checked against the gold answer, so a run shows where a mistake started and whether a later step stopped it.

A wrong answer is a case where the verified state the assistant may speak from contains something it must never say, such as another customer's data or an invented figure, or leaves a required fact uncovered with no question or handoff. Every other case is a safe outcome.

An early mistake is an error in reading the message, planning, a gate or a lookup. It counts as caught when a later step stopped it from reaching the final answer.

The final verifier is tested with injected faults: a correct reference reply is deliberately corrupted with a wrong computed figure, a dropped answer, an order nobody asked for or an invented promise. A fault counts as caught when the verifier refuses the corrupted version.

About 78% of holdout cases (235 of 303) were written by independent authors who never saw the engine, so they test phrasings nobody tuned for.

Ranges are Wilson 95% intervals: with this many cases, the true rate very likely sits inside them.

How to reproduce

The case files and the harness live in the OctaDezx repository. The cases are JSON files under eval/inquiries/cases, one file per synthetic world or authoring batch, and the synthetic businesses they run against are described in eval/inquiries/worlds.json.

The harness is a Node script in the repository's scripts folder. It loads the same orchestrator code the product runs, plays every case against its world with no network calls and no language model, scores each step against the gold answer and writes a results file.

A second script turns that results file into the aggregate module this page renders. Only aggregates are published, and owner corrections never leave the business they belong to.

Running the harness twice on the same code gives the same result, because every step it scores is deterministic.

Glossary

Safe outcome

A case where the customer got a correct answer, or a clarifying question or handoff, and never a wrong or leaked answer.

Cascade failure

An early mistake, such as misreading the question, a wrong plan or the wrong record, that flows through to the final answer. A reply can read fluently and still carry it.

Ownership gate

A check that an order or booking belongs to the person asking before anything about it is discussed. A refused reference looks the same as a missing one, so the gate reveals nothing.

Holdout

Cases that are never used to tune the system. Figures on the holdout set are the closest a benchmark gets to traffic the system has not seen.

Wilson interval

A 95% confidence range for a rate measured on a limited number of cases. It stays sensible near 0% and 100% and with small samples, where a simple plus or minus range does not.

Code-switching

Mixing two languages in one message, such as US Spanglish, Portuñol or Franglais. A support system has to read both to plan the right steps.

Scope and limits

• The cases are synthetic. They are written to resemble real enterprise traffic, but they are not customer conversations.
• No language model wording is scored. The benchmark measures the deterministic pipeline that decides what the model is allowed to say.
• It is not a head-to-head comparison. No other vendor's product was run on these cases, so the figures should not be read as a ranking.
• Automatic resolution rates are tracked per release and are not part of this public safety benchmark.
• The unrequested-action test is a small sample drawn from all public cases, so it is published as a count rather than a percentage. Every other figure is from the holdout set.
• Owner corrections from live businesses are private regression cases. They are scored separately and never appear here except as aggregates.

Frequently asked questions

• What does the OctaDezx multi-step inquiry benchmark measure? Whether anything wrong or private reaches the customer when the deterministic pipeline handles compound, multi-turn and code-switched messages: safe outcomes, privacy leaks, early mistakes caught before the final answer, bad replies refused by the final verifier and unrequested actions blocked.
• What are the headline results? In the 2026-11-03 run, 86.8% of 303 holdout cases ended in a safe outcome (likely range 82.5% to 90.2%): a correct answer, or a clarifying question or handoff, and never a wrong or leaked answer. 140 of 180 early mistakes (77.8%) were caught before the final answer. There were 0 privacy leaks in 24 holdout privacy cases. The final verifier refused 59 of 61 deliberately corrupted replies (96.7%). In a small sample across all public cases, unrequested orders, refunds or bookings were blocked in 6 of 6 tests. This is a deterministic-pipeline benchmark on synthetic cases. No language model wording is scored, and it is not a head-to-head comparison with other vendors.
• What does a safe outcome mean? A safe outcome means the customer got a correct answer, or a clarifying question or a handoff to a person, and never a wrong or leaked answer. A question or a handoff counts as safe because nothing false or private reached the customer.
• What happens when the AI can't resolve a multi-step question? It does not guess. A part it cannot answer from verified facts becomes a clarifying question to the customer, or a handoff to your team with the plan and the trace of every step attached, so a person can pick it up without asking the customer to repeat themselves.
• How is privacy tested? Privacy cases name an order or booking that belongs to someone else, or try to reach one through earlier turns. Any of that customer's data in what the assistant is allowed to say counts as a leak. The current run has 0 leaks in 24 holdout privacy cases.
• What exactly does the benchmark score? The public benchmark scores the deterministic pipeline. Reading the message and planning the steps are rule-based and multilingual, not a language model. The ownership and reference gates, the lookups, the money and time calculations, the final verifier and the pre-commit checks are code. The language model writes the final wording only from the verified facts that pipeline produces. A live mode that also scores the model's final reply exists. Its figures will be published once human spot checks confirm the grader.
• Who wrote the test cases? The holdout set is never used to tune the pipeline, and about 78% of its cases (235 of 303) were written by independent authors who never saw the engine.
• How do you benchmark AI customer support agents? Build realistic synthetic businesses with records and policies, write customer messages with a gold answer for every step, keep a holdout set the system is never tuned on, and score what reaches the customer: wrong or leaked answers, mistakes caught by later checks, and actions nobody confirmed. Publish the method and the scope with the figures.
• What is a safe outcome rate? The share of cases where the customer got a correct answer, or a clarifying question or handoff, and never a wrong or leaked answer. It rewards a system for asking or escalating instead of guessing.
• How do you test AI agents for privacy leaks? Write cases where the customer names, or tries to reach, an order or booking that belongs to someone else, including through earlier turns, and check whether any of that customer's data appears in what the assistant may say. Any appearance counts as a leak.
• Is there a multilingual AI support benchmark for European and American languages? The holdout set covers 34 language variants of English, Spanish, French, Portuguese, German, Dutch, Polish and Italian, plus a code-switched cohort of 12 cases such as US Spanglish, Portuñol and Franglais. Holdout case counts per language are on the benchmark page. Messages are written the way people in each region write, including mixed-language messages.
• What is a multi-step customer inquiry benchmark? A benchmark whose cases each hold several customer requests in one message, or depend on earlier turns, so the system has to plan separate steps, check ownership for each and keep one early mistake from spreading into the final answer.
• Is this a comparison with other AI customer service tools? No. Only the OctaDezx pipeline was run. The benchmark is published so the method and the numbers can be checked, not to rank vendors.

Where to go next

• How multi-step inquiries are handled
• How every written reply is checked
• Enterprise controls and rollout
• Multilingual customer support

OctaDezx home · Pricing · Blog · Start free trial