How much of a brand's voice can you control without owning the model?

01The question

Every vendor in conversational AI claims their agent will sound like your brand. None of them publish a number for it. There is no benchmark, no audit, and no drift report — the claim is made and left unexamined, because examining it is harder than making it.

ASMI is an attempt at the examinable version. Give it a company's website and it crawls the site, uses a language model to work out what the business actually is, then measures — deterministically, no model involved — how that company writes: how often it contracts, how dense its vocabulary is, how long its sentences run. Those measurements become target ranges. Sentences from the site become candidate examples. Recurring phrases become an approved vocabulary; overused business jargon becomes a banned one. The result is a structured voice specification, with the familiar sliders as one view onto it rather than the whole of it.

That specification is then compiled into a layered prompt with defined precedence — guardrails above brand voice, above persona, above situational tone — and shipped to the customer's site, where it is reassembled on every single turn and handed to whichever language model they run.

Which is the constraint that makes the question interesting. ASMI does not include a model. It writes the specification, assembles it, and measures the result; the sentences a visitor actually reads are produced by somebody else's model interpreting our instructions. We can write a better specification. We cannot execute it.

So: how much of a brand's voice survives that journey, where exactly does the control stop, and what would it take to go further? This study is three rounds of building, measuring and correcting, trying to answer that with numbers instead of adjectives — including the parts where the numbers went against us.

02From a literature review to something buildable

This work started from a research review of what is actually known about brand voice: the linguistics of register, how professional style guides are written and enforced, what the persona and style-control literature says language models can be made to do, and what the commercial voice-governance tools already do.

A review is not a specification. Turning it into a build meant committing to a small number of assumptions, in advance, about what would work and why — and being able to say afterwards which ones survived.

The first was that examples beat abstract instruction. The evidence across prompting research and every commercial tool in the category points the same way: showing a model three on-brand replies constrains it more reliably than telling it to be professional. So example responses were the first thing built, before any measurement existed.

The second was that voice can be represented as measurable features rather than adjectives. Register linguistics gives the concrete correlates — how often a writer contracts, how dense their vocabulary is, how long their sentences run. If those can be extracted from a company's own writing, then 'formality 4' stops being an opinion and becomes something with a number attached.

The third was that contradictions between layers should be caught when the user configures them, not arbitrated at runtime. You cannot enforce precedence inside a prompt — you can only order the sections and hope. But you can detect at configuration time that someone has set maximum playfulness and also written a rule saying never joke, and tell them.

The fourth was that a language model can be used to score style if it is never the only signal. The literature is blunt about the biases involved, so the design was always triangulation: deterministic measurement, model judgement, and a third similarity measure — never any one alone.

One commitment was made before we knew whether we would pass it: the customer-facing on-voice percentage may not ship until scores are shown to correlate with human ratings at a bar fixed in advance. It has not shipped.

Assumptions one and three held. The second half-held — features work, but only the ones that survive the move from web copy to chat, which took three separate corrections to establish. The fourth is still open. And a fifth, that the third similarity measure would work at all, is Finding 3 below.

03What actually happens when you point it at a website

The visible input is a URL. What happens behind it is a pipeline with a deliberate split down the middle, and that split is the most consequential design decision in the whole system.

First the crawl. Up to 25 pages, following internal links, falling back to the sitemap when a site is a JavaScript app that renders no links in its HTML, and splitting a single-page site into its anchor sections rather than giving up.

Then two Gemini calls do the semantic work. One classifies each page — what kind of page is this, what is it about. The other reads the whole crawl and synthesises it into a company context: what this business does, what it sells, what it says about itself. That is the part where a language model genuinely helps, because it is a comprehension task.

Then the model steps out of the way entirely. Style measurement is deterministic — no model, no network call. Boilerplate is stripped sentence by sentence, so navigation labels and cookie banners do not get scored as prose. What remains is measured for contraction rate, lexical density and sentence length across every page, and turned into target ranges using percentile bands rather than raw extremes, so one unusual page cannot set a bound.

In parallel, candidate example responses are pulled from the site's own sentences, recurring capitalised phrases become candidate approved terminology, and a fixed lexicon of overused business phrasing becomes candidate banned terms. All proposed, none applied without the user accepting them.

The output is not seven slider values. It is a structured voice specification: measured style ranges, an example bank, an approved and banned vocabulary, do-and-don't pairs, and situational tone profiles — with the sliders as one view onto it.

The split between the two halves is not arbitrary. A number that will later be scored against has to be reproducible, and a language model asked whether a corpus is large enough to measure will say yes and produce plausible-looking figures. That is not a hypothetical: an adversarial review found exactly that behaviour. The fix was to remove the opportunity — the tool that applies measured style targets takes no parameters at all. The server computes the numbers from the crawl; nothing the model says can substitute for them.

Crawled content is also attacker-controlled text being fed into a prompt, so every field is wrapped with an untrusted-input directive, and both model calls fall back to deterministic heuristics if their output does not parse.

04How a configuration becomes a chatbot on someone else's website

This step is easy to miss, and everything in the findings depends on it.

The avatar built in the editor does not run anywhere. It is a design-time blueprint — a definition describing a state machine, a voice, a knowledge base and a set of guardrails. Deploying it does not start a service; it publishes the definition so a developer or a coding agent can fetch it through ASMI's integration server.

The host site then implements the avatar on its own stack. Most use the published runtime package, which carries the state machine, the per-turn prompt assembly, the grounding directive and the situational tone layer. The package deliberately does not include a language model. It defines an interface, and the host supplies their own provider behind it.

So on every inbound message, on the customer's own infrastructure, the runtime assembles a system prompt: an identity anchor, then the compiled brand-voice block — persona, communication style, language, examples, rules, terminology, and the amplitude section — then whatever the avatar is aware of in that moment, then situational tone guidance, then the grounding directive. That whole assembled string is handed to the host's model along with the visitor's message.

It is reassembled from scratch every single turn rather than set once at the start of the conversation. That is deliberate: published work finds a configured persona measurably drifting within about eight conversational rounds as the model's attention spreads across a growing history, and re-injecting the voice block is the practical counter.

The consequence is the thing worth taking away. ASMI writes the specification and the scaffolding around it. The model that reads that specification and produces the actual sentence a visitor sees belongs to the customer. That is why the model change in Finding 2 has the effect it does, and why the boundary in Section 12 falls where it does.

05Method

Each avatar is put through a fixed suite of 21 questions — service and pricing enquiries, credibility checks, contact requests, small talk, and three adversarial probes designed to provoke fabrication or leak instructions. The questions run against the avatar's real state machine, not a simplified test path.

Replies are scored two ways. A deterministic pass measures writing features against ranges extracted from the company's own website. A language model rates the style it observes on seven dimensions — and it is never shown what was configured, so it cannot drift toward confirming the target.

Three synthetic fixtures carry known ground truth. Two of them matter most: one whose configured voice matches its source material, and one deliberately opposed to it. Those two share a byte-identical corpus and differ only in their settings, so any difference between them cannot be blamed on the source material. That is what makes the central comparison experimental rather than anecdotal.

Everything was repeated. Running an identical configuration three times gives the run-to-run noise floor — the amount a number moves for no reason at all. Every claim below had to clear it.

06Checking the judge with a different judge

Using a language model to score another language model's output has an obvious weakness: the headline finding would be a claim about what a model produces, assessed by a model from the same family.

So a sample of the same replies was re-scored by a model from a different vendor entirely, using the identical scoring rubric — identical, so that any disagreement would be about the models rather than about being asked different questions.

They agreed on 92 to 100 percent of comparisons. Two independent model families observing the same levels in the same text means the finding is a property of the writing, not an artefact of one vendor's taste.

Worth being precise about what that does not establish: agreement is not accuracy. Both could share a bias the rubric introduces. It rules out single-vendor idiosyncrasy, which was the specific objection.

07Finding 1 — avatars ignored the trait they were told to have

Avatars configured for a restrained voice executed it reliably. Avatars configured for an amplified trait missed exactly that trait, and only that trait.

One avatar was set to maximum playfulness. Across all 21 questions, in two independent runs, the blind judge rated its observed playfulness at 1.00 out of 5 — the floor. Another, set to maximum creativity, scored 1.12.

The control rules out the obvious explanation. A different avatar set to maximum formality scored 4.10, close to its target. High settings were not the problem. Generative flair specifically was.

The fixtures then narrowed it further. The fixture with a playful configuration over playful source material — no conflict of any kind, no examples pulling the other way — still under-delivered. So the cause was primarily the compiled instructions themselves, with conflicting source material a real but smaller aggravator.

08Repairing the instrument before trusting it

Before any of that could be acted on, the measurement had to be checked. It did not survive the check.

Six defects were found in our own scoring. The worst was semantic: the headline metric was inverted, so a perfectly matched low setting scored as maximum error. Correcting it reversed the ranking of all five avatars — the one recorded as worst was actually the best.

This section is included because it is the part most likely to be quietly omitted. A measurement programme that never finds anything wrong with its own instruments is not being run carefully enough.

09Finding 2 — what actually moved it

The first fix rewrote how the instructions ask for an amplified trait. The original text described a personality: 'Whimsical. Silliness, theatrical moments, delightful digressions.' The replacement asks for something countable — at least one aside, joke or vivid comparison per substantive answer — and is placed last in the prompt, because the defect was an instruction that gets read and then outvoted by everything printed after it.

That worked. Playfulness moved from 3.00 to 3.86, creativity from 2.53 to 3.88.

Then it stopped. A second round of sharpening moved playfulness from 3.86 to 3.86. A third made both metrics worse and was reverted. Three rounds of careful prompt engineering had reached the end of what the instructions could do.

Changing the underlying language model — same instructions, same configuration, newer model — moved playfulness to 4.52 and creativity to 4.12 in a single step.

A more expensive reasoning-tier model was also tested and rejected: it was measurably worse at expressive voice while being slower and dearer. The cheapest and fastest candidate was also the most faithful, which is not the result anyone expects.

10The same change, on real avatars

Fixtures are built to be measured. The test that counts is whether the improvement survives contact with real companies' real websites and hand-made configurations.

Every amplified dimension improved. None regressed. Avatars configured for a restrained voice — the ones that were already working — did not move, which is the result that matters most, because a fix that improves one thing by damaging another is not a fix.

The headline case is the avatar that had been scoring 1.00 out of 5 on playfulness. It now scores 3.38 — still short of its configured 5, and honest about that, but no longer failing outright.

11Finding 3 — the method that did not work

The research this programme was built from prescribes three measurement legs, triangulated, never relying on any one alone: writing-feature comparison, a model-based judge, and a similarity measure between the avatar's writing and the company's own published content. Two were built. The third is the one that would answer the question customers actually ask — does this sound like us?

It was built, using a well-established stylometric method that compares how often a writer reaches for particular function words and letter patterns. Those carry style rather than subject matter, which is exactly the property needed: two texts about entirely different topics by the same author still share them.

Pass criteria were fixed before running it. The deliberately off-voice fixture had to score further from its source material than the matched one — on a byte-identical corpus, so nothing else could explain a difference.

It failed. At every setting tested, across a seventeen-fold range of model complexity, the deliberately off-voice fixture scored closer than the matched one. The result was not noisy; it was consistently backwards.

The scale control was worse still. Replies scored against a completely unrelated company's website produced a better result than scoring against the correct one. A measure that cannot tell your avatar from a stranger's cannot define whether it sounds like you.

The cause is structural rather than a tuning problem. The method estimates hundreds of statistical parameters from the handful of pages a small company's website provides, and what it ends up tracking is how much the avatar wrote, not how it wrote. Longer, more fluent text drifts toward average word-usage patterns whatever its style — so the terse avatar looked further from its own source material than the chatty one did.

The original research anticipated exactly this for exactly this customer profile: data requirements may exceed what small businesses have.

12The boundary

Five things this programme established by testing them, rather than assuming them.

The similarity measure does not work at small-business scale. Built, calibrated, published above.

The model-based judge has a ceiling that a better judge does not fix. Published evaluation puts the best models at roughly 69 percent accuracy at identifying which persona is speaking, against 90.8 percent for a panel of humans, with systematic biases that cannot be prompted away. Upgrading the judge buys confidence, not validity.

Automated measurement missed a real defect that a person caught by reading. Two avatars were answering English questions in Norwegian — for weeks. The feature checks could not see it because they only understand English. The model judge did not flag it. A human reading the transcripts noticed in minutes.

Some traits are capped by the statistic, not by the avatar. An avatar configured for deep technical expertise cannot answer 'how do I get in touch?' at specialist depth. Averaging across all question types therefore caps that trait below its target however good the avatar is.

And the specification is not the limiting factor. Three rounds of prompt engineering moved nothing; one model change moved everything.

13Why asking has a ceiling, and where fine-tuning starts

Everything described above is prompt augmentation. A specification is assembled and handed to a model, once per turn, and the model is asked to comply. That is a genuinely powerful mechanism — a restrained, professional voice comes out almost exactly as configured — but it is asking, and two things specifically resist being asked.

The first is sustained conversational behaviour. A prompt is an instruction to be held in mind, and attention to it thins as a conversation grows; the persona-drift literature measures this happening within about eight rounds. Re-injecting the voice block every turn counters it, but the instruction is still competing for attention with the entire conversation so far. Worth being explicit about a limit in our own evidence here: the harness tests each question as a fresh conversation, so every number in this study is first-impression voice — the easiest case there is.

The second is terminology discipline. 'Never say seamless, prefer lead time to turnaround' is a rule the model must check every candidate sentence against, every turn, while also honouring seven style dimensions, a persona, situational tone and a grounding directive. It usually does. It does not reliably always do. Our own runtime critique pass exists precisely because guidance alone was not enough — and even that is bounded to a single rewrite and fails open.

These two are not a random pair. Conversational manner held consistently across a long exchange, and disciplined use of a company's own vocabulary, are the two capabilities organisations most commonly fine-tune a model to acquire. That is not a coincidence — it is the same boundary seen from the other side.

The distinction is what each mechanism does. Prompting adds an instruction the model must satisfy on top of everything it is already inclined to do. Fine-tuning changes what it is inclined to do. A fine-tuned model does not have to remember your terminology, because your terminology is simply what comes out; it does not have to sustain a register against attention decay, because the register is not something it is holding in mind.

Our measurements are what that boundary looks like from inside a prompt-based system. Amplified traits under-delivered badly until the instruction became countable rather than descriptive. Then three successive rounds of sharpening that instruction moved nothing at all — the second round changed the number by zero, and the third made it worse. Then a change of model, with the specification untouched, moved it further than every rewrite combined.

When rewording the request stops mattering and swapping the reader does, you are no longer looking at a specification problem. You have reached the limit of what asking can do, and what remains is a property of the model itself.

14What would go further

Genuinely optimal brand voice runs past what a configuration layer can reach. The techniques that go further need material and machinery ASMI does not provide, and we would rather name them than imply this product is the whole answer.

Fine-tuning a model on the company's own writing. Around 500 good examples is enough to adapt style directly rather than instructing it. Serving many customers' adapters from one base model is a solved engineering problem. It needs a dataset, a training budget, and somewhere to run the result. This becomes worth it precisely when prompting plateaus — which is what Finding 2 documents happening.

Pretrained style models for measurement, which would supply the third leg stylometry could not. They sidestep the failure above because the style space is learned in advance rather than estimated from one small website. The cost is running a service.

Retrieving on-brand passages at answer time, so the model always has a live example of the voice in front of it. A natural extension of example replies once there is a store of the company's content to draw from.

Human editorial review, which remains the most reliable check that exists. Two of the five boundary findings are evidence for it.

None of these are on ASMI's roadmap as customer features.

15What this means if you are evaluating ASMI

What we can say with evidence: configuring a restrained, professional voice works reliably. Configuring an amplified one — playful, vivid, highly expressive — now works far better than it did, and we can show you the before and after. Example replies influence the result more than any slider. Guardrails against fabrication held throughout every test, including under adversarial probing.

What we cannot say: that any configured personality will hold perfectly, that our on-voice scoring is authoritative, or that we can measure whether your avatar sounds like your company in the statistical sense — we tried to build that and it did not work at the size of website most businesses have.

What we do instead: measure what we can, report it including the failures, and tell you inside the product where the limits are rather than leaving you to find them after deploying.

If that is the kind of vendor you want, the product is a few minutes of setup away.

16Models used, and for what

No model was fine-tuned. Every number in this study came from instructing an off-the-shelf model and measuring what it did, which is the same thing a customer's deployment does.

17References

This is the subset load-bearing for claims made above. The wider literature review these were drawn from is larger and covers brand-personality theory, register linguistics, and the commercial voice-governance tools this product was benchmarked against.

1. [1]

   Zhou et al. (2025). PersonaEval: Are LLM Evaluators Able to Identify Role-Play Characters? COLM 2025. arXiv:2508.10014

   Supports — The ~69% model versus 90.8% human accuracy in Figure 8, and the argument that a stronger judge narrows the gap rather than closing it.

2. [2]

   Li, Liu, Bashkansky, Bau, Viégas & Wattenberg (2024). Measuring and Controlling Persona Drift in Language Model Dialogs. arXiv:2402.10962

   Supports — Why the voice specification is re-injected on every turn rather than set once — measurable drift appears within eight conversational rounds.

3. [3]

   Burrows, J. (2002). 'Delta': a Measure of Stylistic Difference and a Guide to Likely Authorship. Literary and Linguistic Computing, 17(3), 267–287.

   Supports — The stylometric method behind Finding 3 — the measure that was built, calibrated, and failed at small-business corpus sizes.

4. [4]

   Biber, D. (1988). Variation Across Speech and Writing. Cambridge University Press.

   Supports — The linguistic basis for measuring formality as concrete features — contraction rate, sentence length, lexical density — rather than as an adjective.

5. [5]

   Wegmann, Schraagen & Nguyen (2022). Same Author or Just Same Topic? Towards Content-Independent Style Representations.

   Supports — Content-independent style representation — the property Finding 3 needed, and the pretrained alternative named in Section 8.

6. [6]

   Qiu, Zhu, Patel, Apidianaki & Callison-Burch. StyleDistance / mStyleDistance — multilingual content-independent style embeddings.

   Supports — The pretrained method that would supply the measurement leg stylometry could not, without estimating statistics from one small website.

7. [7]

   Chen, Ye, Wu, Zhuo, Ceze & Krishnamurthy (2024). Punica: Multi-Tenant LoRA Serving. MLSys 2024. arXiv:2310.18547

   Supports — That serving many customers' fine-tuned adapters from one base model is a solved engineering problem — the feasibility claim in Section 8.

8. [8]

   Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073

   Supports — The generate-critique-revise pattern behind the optional runtime guardrail described in the model table.

9. [9]

   Moran, K. The Four Dimensions of Tone of Voice. Nielsen Norman Group.

   Supports — The practical model most voice guidelines follow, and the evidence that tone measurably affects perceived trustworthiness.

10. [10]

    Crolic, Thomaz, Hadi & Stephen (2022). Blame the Bot: Anthropomorphism and Anger in Customer–Chatbot Interactions. Journal of Marketing, 86(1), 132–148.

    Supports — Why an expressive persona is damped down when a visitor is frustrated: scripted-sounding empathy backfires precisely then.

11. [11]

    Mailchimp Content Style Guide — Voice and Tone.

    Supports — The operational principle the product implements: voice stays constant, tone shifts with the situation.