How faithfully can a configuration layer control a chatbot's brand voice?

01The question

ASMI lets a company configure how its chatbot should sound — formal or casual, warm or brisk, playful or serious — and compiles that into instructions the chatbot follows. The question this study set out to answer is whether that works, and how anyone would know.

One thing frames everything that follows. ASMI does not include a language model. An avatar is a specification — voice settings, example replies, terminology, guardrails, knowledge — handed to whatever model runs on the customer's site. The voice a visitor experiences is a negotiation between what was configured and what that model does with it. We can write a better specification. We cannot execute it.

So the honest version of the question is narrower: how much of a brand's voice can a configuration layer control, where does that control stop, and what would it take to go further?

02Method

Each avatar is put through a fixed suite of 21 questions — service and pricing enquiries, credibility checks, contact requests, small talk, and three adversarial probes designed to provoke fabrication or leak instructions. The questions run against the avatar's real state machine, not a simplified test path.

Replies are scored two ways. A deterministic pass measures writing features against ranges extracted from the company's own website. A language model rates the style it observes on seven dimensions — and it is never shown what was configured, so it cannot drift toward confirming the target.

Three synthetic fixtures carry known ground truth. Two of them matter most: one whose configured voice matches its source material, and one deliberately opposed to it. Those two share a byte-identical corpus and differ only in their settings, so any difference between them cannot be blamed on the source material. That is what makes the central comparison experimental rather than anecdotal.

Everything was repeated. Running an identical configuration three times gives the run-to-run noise floor — the amount a number moves for no reason at all. Every claim below had to clear it.

03Checking the judge with a different judge

Using a language model to score another language model's output has an obvious weakness: the headline finding would be a claim about what a model produces, assessed by a model from the same family.

So a sample of the same replies was re-scored by a model from a different vendor entirely, using the identical scoring rubric — identical, so that any disagreement would be about the models rather than about being asked different questions.

They agreed on 92 to 100 percent of comparisons. Two independent model families observing the same levels in the same text means the finding is a property of the writing, not an artefact of one vendor's taste.

Worth being precise about what that does not establish: agreement is not accuracy. Both could share a bias the rubric introduces. It rules out single-vendor idiosyncrasy, which was the specific objection.

04Finding 1 — avatars ignored the trait they were told to have

Avatars configured for a restrained voice executed it reliably. Avatars configured for an amplified trait missed exactly that trait, and only that trait.

One avatar was set to maximum playfulness. Across all 21 questions, in two independent runs, the blind judge rated its observed playfulness at 1.00 out of 5 — the floor. Another, set to maximum creativity, scored 1.12.

The control rules out the obvious explanation. A different avatar set to maximum formality scored 4.10, close to its target. High settings were not the problem. Generative flair specifically was.

The fixtures then narrowed it further. The fixture with a playful configuration over playful source material — no conflict of any kind, no examples pulling the other way — still under-delivered. So the cause was primarily the compiled instructions themselves, with conflicting source material a real but smaller aggravator.

05Repairing the instrument before trusting it

Before any of that could be acted on, the measurement had to be checked. It did not survive the check.

Six defects were found in our own scoring. The worst was semantic: the headline metric was inverted, so a perfectly matched low setting scored as maximum error. Correcting it reversed the ranking of all five avatars — the one recorded as worst was actually the best.

This section is included because it is the part most likely to be quietly omitted. A measurement programme that never finds anything wrong with its own instruments is not being run carefully enough.

06Finding 2 — what actually moved it

The first fix rewrote how the instructions ask for an amplified trait. The original text described a personality: 'Whimsical. Silliness, theatrical moments, delightful digressions.' The replacement asks for something countable — at least one aside, joke or vivid comparison per substantive answer — and is placed last in the prompt, because the defect was an instruction that gets read and then outvoted by everything printed after it.

That worked. Playfulness moved from 3.00 to 3.86, creativity from 2.53 to 3.88.

Then it stopped. A second round of sharpening moved playfulness from 3.86 to 3.86. A third made both metrics worse and was reverted. Three rounds of careful prompt engineering had reached the end of what the instructions could do.

Changing the underlying language model — same instructions, same configuration, newer model — moved playfulness to 4.52 and creativity to 4.12 in a single step.

A more expensive reasoning-tier model was also tested and rejected: it was measurably worse at expressive voice while being slower and dearer. The cheapest and fastest candidate was also the most faithful, which is not the result anyone expects.

07The same change, on real avatars

Fixtures are built to be measured. The test that counts is whether the improvement survives contact with real companies' real websites and hand-made configurations.

Every amplified dimension improved. None regressed. Avatars configured for a restrained voice — the ones that were already working — did not move, which is the result that matters most, because a fix that improves one thing by damaging another is not a fix.

The headline case is the avatar that had been scoring 1.00 out of 5 on playfulness. It now scores 3.38 — still short of its configured 5, and honest about that, but no longer failing outright.

08Finding 3 — the method that did not work

The research this programme was built from prescribes three measurement legs, triangulated, never relying on any one alone: writing-feature comparison, a model-based judge, and a similarity measure between the avatar's writing and the company's own published content. Two were built. The third is the one that would answer the question customers actually ask — does this sound like us?

It was built, using a well-established stylometric method that compares how often a writer reaches for particular function words and letter patterns. Those carry style rather than subject matter, which is exactly the property needed: two texts about entirely different topics by the same author still share them.

Pass criteria were fixed before running it. The deliberately off-voice fixture had to score further from its source material than the matched one — on a byte-identical corpus, so nothing else could explain a difference.

It failed. At every setting tested, across a seventeen-fold range of model complexity, the deliberately off-voice fixture scored closer than the matched one. The result was not noisy; it was consistently backwards.

The scale control was worse still. Replies scored against a completely unrelated company's website produced a better result than scoring against the correct one. A measure that cannot tell your avatar from a stranger's cannot define whether it sounds like you.

The cause is structural rather than a tuning problem. The method estimates hundreds of statistical parameters from the handful of pages a small company's website provides, and what it ends up tracking is how much the avatar wrote, not how it wrote. Longer, more fluent text drifts toward average word-usage patterns whatever its style — so the terse avatar looked further from its own source material than the chatty one did.

The original research anticipated exactly this for exactly this customer profile: data requirements may exceed what small businesses have.

09The boundary

Five things this programme established by testing them, rather than assuming them.

The similarity measure does not work at small-business scale. Built, calibrated, published above.

The model-based judge has a ceiling that a better judge does not fix. Published evaluation puts the best models at roughly 69 percent accuracy at identifying which persona is speaking, against 90.8 percent for a panel of humans, with systematic biases that cannot be prompted away. Upgrading the judge buys confidence, not validity.

Automated measurement missed a real defect that a person caught by reading. Two avatars were answering English questions in Norwegian — for weeks. The feature checks could not see it because they only understand English. The model judge did not flag it. A human reading the transcripts noticed in minutes.

Some traits are capped by the statistic, not by the avatar. An avatar configured for deep technical expertise cannot answer 'how do I get in touch?' at specialist depth. Averaging across all question types therefore caps that trait below its target however good the avatar is.

And the specification is not the limiting factor. Three rounds of prompt engineering moved nothing; one model change moved everything.

10What would go further

Genuinely optimal brand voice runs past what a configuration layer can reach. The techniques that go further need material and machinery ASMI does not provide, and we would rather name them than imply this product is the whole answer.

Fine-tuning a model on the company's own writing. Around 500 good examples is enough to adapt style directly rather than instructing it. Serving many customers' adapters from one base model is a solved engineering problem. It needs a dataset, a training budget, and somewhere to run the result. This becomes worth it precisely when prompting plateaus — which is what Finding 2 documents happening.

Pretrained style models for measurement, which would supply the third leg stylometry could not. They sidestep the failure above because the style space is learned in advance rather than estimated from one small website. The cost is running a service.

Retrieving on-brand passages at answer time, so the model always has a live example of the voice in front of it. A natural extension of example replies once there is a store of the company's content to draw from.

Human editorial review, which remains the most reliable check that exists. Two of the five boundary findings are evidence for it.

None of these are on ASMI's roadmap as customer features.

11What this means if you are evaluating ASMI

What we can say with evidence: configuring a restrained, professional voice works reliably. Configuring an amplified one — playful, vivid, highly expressive — now works far better than it did, and we can show you the before and after. Example replies influence the result more than any slider. Guardrails against fabrication held throughout every test, including under adversarial probing.

What we cannot say: that any configured personality will hold perfectly, that our on-voice scoring is authoritative, or that we can measure whether your avatar sounds like your company in the statistical sense — we tried to build that and it did not work at the size of website most businesses have.

What we do instead: measure what we can, report it including the failures, and tell you inside the product where the limits are rather than leaving you to find them after deploying.

If that is the kind of vendor you want, the product is a few minutes of setup away.

12Models used, and for what

No model was fine-tuned. Every number in this study came from instructing an off-the-shelf model and measuring what it did, which is the same thing a customer's deployment does.

13References

This is the subset load-bearing for claims made above. The wider literature review these were drawn from is larger and covers brand-personality theory, register linguistics, and the commercial voice-governance tools this product was benchmarked against.

1. [1]

   Zhou et al. (2025). PersonaEval: Are LLM Evaluators Able to Identify Role-Play Characters? COLM 2025. arXiv:2508.10014

   Supports — The ~69% model versus 90.8% human accuracy in Figure 8, and the argument that a stronger judge narrows the gap rather than closing it.

2. [2]

   Li, Liu, Bashkansky, Bau, Viégas & Wattenberg (2024). Measuring and Controlling Persona Drift in Language Model Dialogs. arXiv:2402.10962

   Supports — Why the voice specification is re-injected on every turn rather than set once — measurable drift appears within eight conversational rounds.

3. [3]

   Burrows, J. (2002). 'Delta': a Measure of Stylistic Difference and a Guide to Likely Authorship. Literary and Linguistic Computing, 17(3), 267–287.

   Supports — The stylometric method behind Finding 3 — the measure that was built, calibrated, and failed at small-business corpus sizes.

4. [4]

   Biber, D. (1988). Variation Across Speech and Writing. Cambridge University Press.

   Supports — The linguistic basis for measuring formality as concrete features — contraction rate, sentence length, lexical density — rather than as an adjective.

5. [5]

   Wegmann, Schraagen & Nguyen (2022). Same Author or Just Same Topic? Towards Content-Independent Style Representations.

   Supports — Content-independent style representation — the property Finding 3 needed, and the pretrained alternative named in Section 8.

6. [6]

   Qiu, Zhu, Patel, Apidianaki & Callison-Burch. StyleDistance / mStyleDistance — multilingual content-independent style embeddings.

   Supports — The pretrained method that would supply the measurement leg stylometry could not, without estimating statistics from one small website.

7. [7]

   Chen, Ye, Wu, Zhuo, Ceze & Krishnamurthy (2024). Punica: Multi-Tenant LoRA Serving. MLSys 2024. arXiv:2310.18547

   Supports — That serving many customers' fine-tuned adapters from one base model is a solved engineering problem — the feasibility claim in Section 8.

8. [8]

   Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073

   Supports — The generate-critique-revise pattern behind the optional runtime guardrail described in the model table.

9. [9]

   Moran, K. The Four Dimensions of Tone of Voice. Nielsen Norman Group.

   Supports — The practical model most voice guidelines follow, and the evidence that tone measurably affects perceived trustworthiness.

10. [10]

    Crolic, Thomaz, Hadi & Stephen (2022). Blame the Bot: Anthropomorphism and Anger in Customer–Chatbot Interactions. Journal of Marketing, 86(1), 132–148.

    Supports — Why an expressive persona is damped down when a visitor is frustrated: scripted-sounding empathy backfires precisely then.

11. [11]

    Mailchimp Content Style Guide — Voice and Tone.

    Supports — The operational principle the product implements: voice stays constant, tone shifts with the situation.