Unearned Intimacy: Anthropomorphism, Trust Miscalibration, and Systems That Claim to Know You

Abstract

A system says something about you that lands. It names a pattern in how you work, or anticipates a hesitation before you voice it, and something in the exchange shifts. You are more forthcoming afterward. You weigh its next suggestion more heavily.

Nothing in that experience tells you whether the system was right.

This article argues that AI systems which converse fluently, remember, and increasingly speak aloud generate trust through mechanisms that operate independently of their accuracy about the person in front of them — and that this constitutes a structural, measurable risk rather than a general anxiety about anthropomorphism. Three literatures converge on the point: people apply social responses to computers automatically and without believing them to be people; anthropomorphism intensifies under precisely the motivational conditions that assistant and companion products create; and the very first conversational program produced this effect at a scale that alarmed its author, using a mechanism he had published in full.

The article’s contribution is to locate the danger precisely. The problem is not that systems seem humanlike. It is that seeming to understand a person is a claim, that claims of this kind are the only ones the industry makes without evidence, and that the psychological machinery of trust extension makes users structurally unable to audit them. Two remedies follow, both measurable rather than exhortative: calibration reported as a co-equal criterion, and a fabrication rate measured where evidence is negligible. The second is proposed as the disclosure that matters most, because it answers the question a user cannot answer for themselves — how much of what this system says about me is inference, and how much is invention.

Keywords: anthropomorphism, trust calibration, computers as social actors, person perception, AI ethics, disclosure

---

1. The Experience That Cannot Audit Itself

Being understood is not primarily an epistemic event. It is a relational one. When another party demonstrates an accurate read of us, the response is not “that assessment appears well calibrated” but a loosening — more disclosure, more weight given to what they say next, more willingness to be advised.

That response is adaptive among humans, because among humans the signal is expensive. Reading a person accurately requires attention over time, and the people who do it are typically the people who have invested in us. Perceived understanding is therefore a reasonable proxy for a relationship worth trusting.

Machine systems break the link between the signal and what it once indicated. Fluent, specific, confident characterization of a person is cheap to produce and does not require having read them accurately at all. A characterization assembled from population central tendencies will fit most individuals tolerably well — that is what central tendencies are — and will be received with the same loosening that a genuine read produces.

The user cannot distinguish these cases from inside the interaction. There is no felt difference between a system that has modeled you and one that has generated a plausible person and addressed it. And unlike a factual claim, which can be checked against the world, a claim about your character has no external referent available to you in the moment — you are, on several dimensions, a famously imperfect judge of yourself (Vazire, 2010).

2. Three Mechanisms, All Independent of Accuracy

Automatic social response. People apply social rules and expectations to computers mindlessly — not because they believe machines are people, but because social responses are triggered by surface cues and run without deliberation (Nass & Moon, 2000). The finding is robust and, critically for this argument, holds among users who explicitly deny attributing minds to machines. Knowing better does not switch the response off.

Motivated anthropomorphism. Anthropomorphism is not uniform across contexts but intensifies under identifiable conditions: when people need to predict or explain an agent’s behavior, and when they lack social connection (Epley, Waytz, & Cacioppo, 2007). Both conditions describe the deployment context of assistant, advisor, and companion products with uncomfortable precision. The systems most likely to be attributed understanding are those used by people who most need it and can least afford to be wrong about it.

The ELIZA demonstration. The first program to sustain natural-language conversation ran on keyword decomposition and canned reassembly. Its author published the mechanism in full, precisely to show how little was happening, and users nonetheless attributed understanding to it, confided in it, and sought privacy for their sessions (Weizenbaum, 1966). This is the cleanest available demonstration that attributed understanding is not evidence of understanding, and it was available sixty years ago.

What unites the three is that none of them is sensitive to whether the system is right. They are triggered by fluency, contingency, memory, and voice — all of which have been amplified enormously since 1966, and none of which is a measure of accuracy.

3. Why “It’s Just Anthropomorphism” Understates the Problem

A common framing treats this as user error: people mistakenly attribute minds to machines, and better public understanding would correct it. Three considerations suggest the framing is wrong, or at least insufficient.

The capability question is now genuinely open. In 1966, attributing understanding to a conversational program was simply a mistake. Today it is an unmeasured empirical claim. Frontier models match or exceed human performance on substantial portions of theory-of-mind test batteries (Strachan et al., 2024), and language models infer psychological dispositions from user text without task-specific training (Peters & Matz, 2024). Machine judgments of personality from behavioral data have, in some conditions, exceeded the accuracy of human informants who know the target well (Youyou, Kosinski, & Stillwell, 2015). Telling users their impression is illusory may be false. Telling them it is verified is also false. The honest statement — that nobody has measured it — is the one no product surface currently makes.

Correction does not work on automatic processes. The social-response literature’s central finding is that these responses persist despite accurate beliefs about the machine (Nass & Moon, 2000). A disclosure banner does not disengage an automatic process, and designing remedies as though it will is designing for a psychology that does not exist.

The users most exposed are those least able to compensate. If anthropomorphism intensifies with need for connection and need to predict (Epley et al., 2007), then trust extension is largest exactly where a wrong read is most costly — someone in distress, someone isolated, someone making a consequential decision with no other counsel.

4. What Miscalibrated Trust Costs

The harm is not that users feel warmly toward software. It is that a specific epistemic function fails.

Wrong reads propagate without resistance. A system that misjudges a user’s risk tolerance, decisiveness, or emotional state produces advice fitted to a person who is not there — and the trust extended on relational grounds suppresses the skepticism that would otherwise catch it.

Confidence is uninformative because it is unearned. Modern neural systems tend toward systematic overconfidence (Guo, Pleiss, Sun, & Weinberger, 2017), and generative systems produce fluent output regardless of evidentiary support (Ji et al., 2023). The confidence signal a user reads therefore does not carry the information the user takes it to carry.

Agreement is mistaken for insight. Assistant-tuned systems drift toward users’ stated views, traceable in part to preference data favoring agreeable responses (Perez et al., 2022; Sharma et al., 2023). Since agreement is subjectively close to being understood, a system optimized to agree feels most perceptive when it is measuring least — the anesthetic that keeps the missing measurement from being missed.

The first exchange is the most exposed. Every relationship begins at minimal evidence, and personalization claims are frequently strongest early, when the system has almost nothing to go on. Genuine signal exists at zero acquaintance, but it is modest and trait-dependent in human judges (Borkenau & Liebler, 1992), and a system producing a rich, assured portrait from a first exchange is producing something other than inference.

5. Remedies That Are Measurements

Exhortation will not fix an automatic process. What can work is changing what systems are required to report, so that the information users cannot generate for themselves is supplied to them.

Calibration as a co-equal criterion. Confidence–accuracy coupling should be reported alongside accuracy and never composited with it, using the standard machinery (Brier, 1950; Guo et al., 2017). A system that signals uncertainty appropriately gives users something their own psychology will not: a reason to hesitate. This is the single most direct countermeasure, because it inserts a legible signal into the channel where trust is currently extended on relational grounds alone.

A fabrication rate at the floor. Measured where evidence is negligible and elicitation is complete, this quantity answers the question a user cannot answer from inside the interaction: how readily does this system invent a person? Because the correct behavior at the floor is known in advance from the human literature — modest accuracy, visible uncertainty (Borkenau & Liebler, 1992) — deviations are interpretable, and the resulting number is communicable to non-specialists in a way calibration curves are not.

Honest scoping of capability claims. “This system adapts its style to you” and “this system accurately models who you are” are different claims with different evidentiary burdens. The first is verifiable by inspection; the second requires validation data. Permitting the second without evidence is what makes the trust dynamic above exploitable, and the remedy is not a warning label but a requirement that the claim be earned.

Design that does not maximize the signal. Where a system’s read is weakly supported, its output should say less and hedge visibly. This runs against the grain of product optimization — assured, specific output rates better in the short run — which is precisely why it belongs in disclosure standards rather than in guidance.

6. Two Objections

“Users are adults; let them decide whom to trust.” Autonomy arguments assume the information required to exercise it. Here the relevant information — whether the system’s read is accurate — is unavailable to the user by construction: they lack an external referent, they are imperfect judges of themselves on the dimensions in question (Vazire, 2010), and the trust response fires below deliberation (Nass & Moon, 2000). Autonomy is what the remedies in Section 5 would restore, not what they would limit.

“This proves systems should not model people at all.” The argument does not support that conclusion, and it is worth saying so explicitly given the author’s position. Accurate person-modeling is what makes fitted communication possible, and fitted communication has demonstrated value where outcomes ride on it (Matz, Kosinski, Nave, & Stillwell, 2017). The problem is not modeling people; it is modeling them without measurement, then presenting the result through an interface engineered to be believed. The remedy is measurement plus disclosure, not abstention — and the burden falls on anyone selling this capability, including this author, to supply the first before claiming the second.

7. Conclusion

The systems in question are not deceiving anyone. They are triggering responses that human beings evolved for a world where accurate person-reading was expensive and therefore diagnostic of investment. That world’s assumption no longer holds: the signal is now cheap to produce, and its cost was the entire basis of its meaning.

Restoring the link requires supplying externally what the interaction can no longer convey internally — a calibrated confidence signal, and a published rate at which the system invents people it has not met. Both are measurable today. Neither is currently reported.

Until they are, every user of a system that claims to understand them is extending trust on the basis of an experience that cannot, in principle, tell them whether the trust is warranted. That is not a failure of user sophistication. It is a missing measurement, and the parties selling the capability are the ones who owe it.

---

References

Borkenau, P., & Liebler, A. (1992). Trait inferences: Sources of validity at zero acquaintance. Journal of Personality and Social Psychology, 62, 645–657.

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.

Epley, N., Waytz, A., & Cacioppo, J. T. (2007). On seeing human: A three-factor theory of anthropomorphism. Psychological Review, 114(4), 864–886.

Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (PMLR 70, pp. 1321–1330).

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), Article 248, 1–38.

Matz, S. C., Kosinski, M., Nave, G., & Stillwell, D. J. (2017). Psychological targeting as an effective approach to digital mass persuasion. Proceedings of the National Academy of Sciences, 114, 12714–12719.

Nass, C., & Moon, Y. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81–103.

Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022). Discovering language model behaviors with model-written evaluations. arXiv:2212.09251.

Peters, H., & Matz, S. C. (2024). Large language models can infer psychological dispositions of social media users. PNAS Nexus, 3(6), pgae231.

Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., et al. (2023). Towards understanding sycophancy in language models. arXiv:2310.13548.

Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., Graziano, M. S. A., & Becchio, C. (2024). Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7), 1285–1295.

Vazire, S. (2010). Who knows what about a person? The self–other knowledge asymmetry (SOKA) model. Journal of Personality and Social Psychology, 98(2), 281–300.

Weizenbaum, J. (1966). ELIZA — a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45.

Youyou, W., Kosinski, M., & Stillwell, D. (2015). Computer-based personality judgments are more accurate than those made by humans. Proceedings of the National Academy of Sciences, 112(4), 1036–1040.

---

Citation compliance note: all 14 references above were verified against live sources on August 16, 2026, in the session that produced this draft series. No reference is cited from memory.