Why making AI tools feel human is a behavioural and governance decision, not a harmless branding choice.

Right now, companies are maturing in the AI race and are moving quickly from experimenting with general AI to buying/ developing products and embedding them into real workflows. As part of the issue around getting recalcitrant employees to use those tools, we use psychological persuasion to aid adoption and today am casting my beady eye to examine the next pitfall emerging from adoption data: over reliance. Imagine now that your employees overly trust output and fail to check it, amplifying corporate risk.

As part of the adoption piece, one clever strategy is to make the technology feel familiar and give it a human name, add a cutesy avatar, let it chat back in the first person. Launch it in a way that ascribes trust; describe it as a colleague, teammate or digital employee. Perhaps even place it on the organisation chart.

It sounds like harmless branding with a dollop of psychological safety to encourage use, but this is a dangerous road.

Organisations are not simply making AI easier to use; to ensure ROI, they need to make it easier to trust. Unfortunately, that can alter expectations, scrutiny and responsibility without changing the underlying capability of the product at all.

The product and the story around the product

An AI product arrives with technical capabilities, limitations and risks. Organisations alsoadd a second layer: the identity, language and behaviours through which employees experience it.

That layer matters because people respond not only to what a system is, but to what they believe it is. Kahneman's distinction between fast, intuitive System 1 thinking and slower, deliberate System 2 thinking provides a useful lens. Human names, faces, fluent language and confident answers can activate familiar social shortcuts before deliberate evaluation begins.1

The framing effect means identical information can produce different decisions depending on its presentation. The halo effect allows one positive feature, such as warmth or fluency, to influence our wider judgement of competence. Cognitive ease makes information that is simple and familiar feel more credible. All of us use these communication short cuts with people everyday, but the output from AI is so consistently smooth we trust it even more.

That is the crux of the problem, a warm interface can change the perception of the tool without improving its reliability

What happens when the tool becomes "Alex"?

Research led by Boston University professor Emma Wiles tested employee framing directly. Managers reviewed identical documents containing built-in errors. The source was described as an AI tool, an AI employee called ALEX-3, or a human employee called Alex.2

Across the full sample, average framing effects were small. But among managers whose organisations had already placed AI agents on their charts, describing the same work as produced by an AI employee resulted in weaker review, fewer errors detected, more escalation and a shift in perceived accountability away from the manager.

The evidence is still coming in and there is correlation but not proof that a name alone causes overtrust. A 2026 workplace study of AI-drafted emails found that superficial human-like design had little effect on reliance. Perceived agency and previous AI experience mattered more.3 Also, the effect is not culturally uniform. In the study, participants in Brazil, Egypt, Nigeria, India, Mexico and Indonesia rated the same chatbot as considerably more human-like than participants in the United States, Germany, Japan and South Korea.

But perceiving an AI as human-like is not the same as trusting it. The researchers therefore tested a second question: when they deliberately made the chatbot more human-like, did people trust it more?

Across the sample as a whole, the answer was no. Human-like design increased anthropomorphism but produced no measurable increase in behavioural trust. Culture did affect some self-reported responses: trust increased among Brazilian participants, decreased among Japanese participants under one design condition, and showed no reliable change in the other countries tested.4

That strengthens the case for governance. Personification is an unpredictable behavioural intervention, not a reliably benign adoption technique.

Easy to trust is not the same as trustworthy

That finding should not be read as reassurance. It shows that surface humanisation is not, by itself, a consistent route to trust. The risk may also sit in the interaction and in the confidence users develop in the system.

A CHI 2025 study surveyed 319 GenAI-using knowledge workers recruited internationally through Prolific. No country dominated the sample: the United Kingdom was the largest reported national group at just 11.6%, while the five largest groups together accounted for 36.7%. Participants contributed 936 examples of using GenAI in real workplace tasks.5

Across those examples, greater confidence in GenAI was associated with less critical thinking, while greater confidence in the worker’s own ability was associated with more. As AI took on more production, human effort shifted towards verification, integration and oversight.

This was an observational, self-reported survey. It cannot establish that confidence in GenAI caused the reduction in critical thinking, and it did not test whether the relationship differed between countries. Nevertheless, the pattern resembles the attentional mechanism behind automation bias: scrutiny can decline as confidence in the automated system rises.6

More recent evidence on sycophancy identifies a more direct interactional risk. Across 11 models and three preregistered experiments involving 2,405 people, AI affirmed users’ actions 49% more often than humans. A single sycophantic interaction reduced participants’ willingness to accept responsibility and repair interpersonal conflict, while increasing their conviction that they were right. Yet participants also trusted and preferred the agreeable AI.7‍ ‍

Thus the sycophancy paradox. The same behaviour that distorted judgement also increased trust and preference. For organisations, that creates a dangerous incentive: engagement and satisfaction measures may reward the very interactions that reduce challenge, accountability and self-correction.

The distinction matters. Human-like presentation does not automatically produce trust, but particular conversational behaviours can.8,9 Agreement and affirmation can make the interaction feel better while weakening the reflection, challenge and accountability taking place inside it.

The design objective should not be maximum trust or maximum adoption. It should be calibrated trust.

THE CALIBRATED-TRUST MODEL

Reliance should rise only when demonstrated capability rises.

Design for clarity, not character

People will resist cumbersome tools and work around products that make their jobs harder. But anthropomorphism is not a substitute for good product and workflow design. Organisations can make AI accessible without turning it into a fictional colleague.

1. Instead of naming the persona, identify and name the function.Use Bid Review Agent or Policy Checking Service, not Ian from Operations.

2. Keep human ownership visible. Every AI-enabled service needs an accountable business owner for its objective, permissions, controls and outcomes.

3. Define the authority boundary. State what the system may access, recommend, decide, execute and escalate.

4. Set review by consequence. Human oversight should reflect risk, reversibility and potential harm, not the confidence or fluency of the output.

5. Preserve useful friction. At critical decisions, make users pause, verify evidence and actively accept responsibility.

6. Measure behaviour, not only accuracy. Track over-reliance, correction ownership, escalation, downstream errors and whether time is saved across the whole system.

AI can be a powerful delegated tool. It can draft, retrieve, recommend, coordinate and increasingly act. But it is not an independent organisational actor and it cannot carry accountability.

If an AI product needs a personality to persuade people to use it, the product and workflow design may not be finished.