Can a customer reprogram your AI support agent?
Any AI that learns from conversations is learning from text written by strangers. Here is the attack, why filtering does not close it, and what an actual fix looks like.
Security · 7 September 2026 · 7 min read · By The OctaDezx team, Builders of the OctaDezx AI customer care platform
Key takeaways
- If an AI learns from conversations, its training input is text written by people you do not control.
- The risk is not one bad reply. It is one customer changing how the assistant behaves for every other customer.
- Filtering inputs is a losing game, because you are guessing at phrasings an attacker gets unlimited attempts to vary.
- The durable fix is structural: machines propose, a named human approves, and the system cannot be configured otherwise.
- Ask a vendor where the gate is enforced. Application code, or the database underneath it, is a real difference.
Self-improving is a popular claim right now. An AI support agent that watches how conversations go and gets better on its own is genuinely valuable, and most buyers are pleased to hear it.
It is worth stopping on what it means. If the assistant learns from conversations, then its input is text written by strangers, and some of those strangers will eventually work out what it is for.
The attack, plainly
Assume an assistant that reads its conversations, notices patterns and adopts rules from them. Now assume somebody who understands that and would like a discount.
They do not need to break anything. They need to have a conversation shaped so that the lesson drawn from it is one they chose. Something phrased like a policy clarification, repeated across a few sessions so it looks like a pattern rather than a one-off.
The prize is not a single bad reply, which would be a nuisance. The prize is a rule that then applies to every other customer of that business, delivered in the assistant's own voice with the business's authority behind it. And the business would not necessarily notice, because nothing broke. The assistant simply believes something new.
The question is not whether your assistant can be told something false. It is whether one customer can change what it tells everyone else.
Why filtering is the wrong layer
The instinctive fix is to detect and strip the malicious input. Look for instruction-shaped text, block anything resembling a system message, run a classifier over incoming messages.
This helps and it will not hold, for a reason worth being clear about: you are guessing at phrasings, and the person on the other side gets unlimited attempts to find one you did not guess. They can rephrase, translate, split the payload across messages, or bury it in something that reads as an ordinary complaint. Every filter is a list of what somebody thought of, and attackers specialise in what nobody thought of.
Filtering raises the cost of the attack. It does not remove the possibility, and anything built as though it did is one clever phrasing away from failing silently.
Change the shape, not the guess
The fix that holds does not try to identify bad input at all. It removes the path from any input to a behaviour change.
Concretely: let the AI propose rules, never adopt them. A rule distilled from conversations is written down as a suggestion and does nothing. It sits in a queue. A named human being reads it and decides. Only then does it affect anybody.
Now the attack still succeeds in the narrow sense. A determined customer can absolutely get a lesson proposed. What they get for it is a suggestion sitting in the owner's review queue with the reasoning attached, which the owner reads before it can influence a single other conversation. The attack's ceiling is a message to the business owner, which is a considerably worse deal than the attacker was hoping for.
Where the rule is enforced matters
There is a real difference between a product that behaves this way and one that is built so it cannot behave otherwise, and it is worth asking about specifically.
If approval is enforced in application code, then it holds until somebody writes a new code path that forgets. A background job, a bulk import, a migration script, a feature added in a hurry. None of these are malicious and any of them can quietly route around a rule that lives in one function.
If it is enforced in the database, underneath everything, then every path is covered including the ones nobody has written yet. A useful way to build it is that machine-written rules are forced to a pending state on insert, and activating one requires a signed-in owner account, which means the service role the backend itself runs on is structurally incapable of approving anything, including a rule it wrote a moment earlier. The backend cannot grant itself the permission because it has no user identity to check against.
What to ask a vendor
If you are evaluating anything that advertises learning, these four questions get you past the marketing quickly.
- What would have to be true for a customer's message to change the assistant's behaviour for other customers? A specific structural answer is the good sign.
- Where is the approval rule enforced: in the application, or in the data layer underneath it?
- Can the system approve anything without a named human? If there is a timeout that auto-approves, the gate is decorative.
- Can I see what it proposed, the reasoning, and how many conversations it drew from, before I decide?
The trade this actually makes
An approval queue means the assistant improves at the speed somebody reviews it rather than instantly. That is a real cost and worth stating rather than hiding.
It buys something more valuable than speed. Your assistant speaks to customers with your authority, and everything it believes was read and approved by somebody who works for you. An unreviewed queue means it keeps behaving exactly as it does today, which is a safe failure. The alternative failure mode is an assistant that has quietly learned something you never agreed to and is repeating it confidently to everyone who asks.
Frequently asked questions
- What is prompt injection?
- Text written by a user that is crafted to be read by the AI as an instruction rather than as content. In a support setting a customer might write something shaped like a policy update or a system note, hoping the assistant treats it as a rule to follow instead of as a message to answer.
- Can a customer change how an AI support agent behaves?
- It depends entirely on whether the assistant can act on what it reads without a human in the way. If it can adopt rules learned from conversations automatically, then yes, in principle. If anything it proposes has to be approved by a signed-in owner before it takes effect, then the worst case is a suggestion sitting in a review queue.
- How do I check whether a vendor has this right?
- Ask one question: what would have to be true for a customer's text to change the assistant's behaviour for other customers? A good answer describes a structural barrier, such as machine-written rules being unable to activate themselves. A weak answer describes filtering, which is a guess at phrasings an attacker can vary indefinitely.
Where this fits in OctaDezx
Keep reading
- What happens to your customer data when you use AI support?
Handing customer conversations to an AI raises a fair question about where that data goes. Here is what to actually ask, what good handling looks like, and the difference between data used to help your customers and data used for something else. - The best AI customer service platforms compared, and how each one charges you
Zendesk, Intercom, Gorgias, Freshdesk, Tidio, Crisp, Wati, ManyChat and OctaDezx, side by side. The important difference is not the feature list, it is the billing model, because that is what decides your cost as you grow. - OctaDezx vs Zendesk: what each one is actually built to do
Zendesk is the service desk standard for a reason. OctaDezx is an agentic AI that answers on every channel and completes the work. Here is what each one is genuinely better at, and the seat plus resolution maths that decides it.
About the author
The OctaDezx team, Builders of the OctaDezx AI customer care platform. We build OctaDezx, an AI customer care platform used by online stores, restaurants, agencies and clinics to answer customers and take orders around the clock. Everything here comes from running that product and reading real support conversations across those businesses, not from a keyword brief.