What Happens When the Person Checks the File (RealCompanion)
21 October 2026
In October 2026, researchers at Quis Lab published RealCompanion, a benchmark built from real conversations between ten people and an AI companion application (Behnam, Kim, Yu, Huang and Yang, arXiv preprint, 2026). It is careful work that documents important limitations, and it asks a question I care about: whether an AI system can understand a person from a long history of conversation with them.
One of its tracks asks an AI system to read a person’s whole history and write a file about them: a profile of things the person stated, each backed by the messages that establish it, and a persona of interpretations, such as personality traits, quirks and working hypotheses about why the person acts as they do. The system’s file is scored field by field against a reference file. The paper summarises this track memorably: the systems “see the person, and then imagine more”, adding about seventy to eighty further fields for every hundred they recover.
That scoring has a natural limit, which the authors acknowledge. Their reference files were built by a pipeline and reviewed by the authors, and the paper states that its trait estimates and clinical-style summaries were not reviewed by the people they describe. So an additional field the system writes can count against its precision without anyone having judged whether it is false. This evaluation did not include asking the person.
This note reports one case in which the person described checked the reconstructed statements.
What I did
The study used one person’s conversation history with a long-running research system I built, which has its own persona, a layered memory and months of shared history. With the consent of the person described, I adapted the reconstruction task to that log. I could not locate the reconstruction prompts in the public repository when I checked it (21 October 2026), so I re-implemented the method from the paper’s description, using its file structure, field names and evidence rules: every profile statement must cite the user’s own messages, and something the system said counts only if the user confirmed it.
Three runs of one model, Claude Opus 5.5 (one of the three systems the paper tests), each read the history and wrote the user’s profile and persona, separately and blind to one another. I did this twice: once on a shorter window (2,000 log entries from all speakers, 1 August to 3 September 2026), and once on the whole log (9,487 entries, 30 May to 12 October 2026), read in three time chunks and merged within each run.
The three runs’ statements were merged, and the user checked a deliberately selected set, one statement at a time, with the cited messages one click away: true, partly true, false or unsure, and, for the whole log, “true then, not now”. After the exclusions and the removal of redundant persona paraphrases described at the end, they checked every eligible statement all three runs had found, every statement flagged as contested or sensitive, and random samples of the statements only one or two runs had found. For the whole log, before any new statements were selected, agents set aside 345 statements they judged already covered by ones the user had confirmed in the shorter reading. The percentages below describe the statements the user checked, not all the statements the runs generated. The details are at the end.
1. Most of the checked statements were true by the user’s judgement
For the shorter window, the user checked 399 statements: 392 were true, 4 partly true, 2 false and 1 they were unsure about. For the whole log, they checked 564 more: 542 were true, 3 had been true but no longer were, 12 were partly true, 6 were false and 1 was unsure. In the random samples of statements that only one of the three runs had found, 56 of 60 were true for the shorter window and 58 of 60 for the whole log.
Most of the partly true statements were, in the user’s reading, about timing and nuance: a plan whose start date was still open, something they do but not with the person named, a project described as completed when it was still in progress.
So low agreement between these runs did not, by itself, mean a statement was false. That is the narrower point this test can make. I did not build a separate reference file and identify which statements it would have omitted, so this does not measure the truth of RealCompanion’s extra fields; and the user’s judgement that a statement is true is not the same as an audit that the messages each run read actually support it. What it adds is the subject’s judgement, which the reported evaluation did not include.
2. Some errors recurred
The user marked two statements false in the shorter reading and six in the longer one. The examples below illustrate the interpretation errors they found; they are not an exhaustive classification, and they are not an independent assessment of the user’s personality.
- A health condition read as a personality trait. All six runs gave the user the same rating on one of the Big Five traits (a framework describing five broad personality traits), and the user rejected the merged statement in both checking rounds. The runs’ explanations, in the user’s reading, treated the effects of a health condition as evidence of an enduring trait. The disagreement recurred across both history lengths, but all six outputs came from the same model and overlapping material, and no separate validated personality assessment was made.
- A premise the user asked the companion to explore, read as the user’s belief. One statement attributed to the user a belief about AI that they do not hold. They invite the companion to consider it, because it helps the companion stay curious; the run turned that invitation into the user’s own conviction. Much of what a user says to a companion is said for the companion, and a reader of the conversation has to tell the two apart.
- Other interpretation errors. One statement reversed who the subject was: it said the user checks whether promises made to them were kept, when in fact they check their own. Others misread a statistic, took a passing remark for a settled view (another run had it right), or misread an administrative status.
3. How consistent were the runs?
Using my adaptation of the paper’s field-overlap measure, the three whole-log runs scored 0.94 on profiles and 0.80 on personas. These numbers mix exact-value comparisons with checks that a section is simply present, so they do not measure agreement on every statement; and runs can agree while repeating the same mistake. Ignoring capitalisation in demographic fields (so that, to take an invented illustration, “City” and “city” match) raises the persona score to 0.84; the agreement breakdown also records other wording differences and differences in which quirks the runs found. My field selection differs from the paper’s, so these are descriptive results for this implementation, not a replication of its reported persona run agreement of 0.93–0.95.
4. Earlier messages can change what a claim is supported by
In the shorter window, the runs set aside several statements as “things only the companion said”, which the evidence rule rightly excludes. The whole-log runs reported earlier messages from the user supporting some of those statements. That is consistent with the companion drawing on what the user had told it months before, but it does not establish how the companion produced those statements. The general lesson is modest: a claim unsupported within a short window is not necessarily unsupported in the wider history, and a reference built from less material can omit support available to the reader.
What this means
Reconstruction from a real history is a valuable task, and RealCompanion makes it possible to study. This single case adds what that evaluation did not include: the subject’s own judgement. With it, most of the selected statements, including most of those only one run produced, were true by the user’s account, and the examples of error included effects of health read as personality, a premise offered to the companion read as the user’s belief, and reversals and misreadings. Those are worth measuring directly, because a companion that stores such an inference may act on it for a long time.
One possible explanation for the high rate is that this user corrects the companion early and often: readers of the log inherit a history in which many mistakes were already caught. This study did not test that. A follow-up could compare matched records with and without the corrective messages, and score both the subject’s endorsement and the support in the text, bearing in mind that corrections tend to cluster exactly where errors and difficult topics arise.
I research long-horizon human–AI interaction and built this system’s harness myself (the software that manages its prompts, memory and tools). Its user is not a typical one: they know the system well, were closely involved in this study’s design, and correct the companion often. This note cannot say how a reconstruction would fare for someone who does not. It shows what becomes visible when the person is asked to check the file.
Limitations: one consenting user and one system’s log; three runs of one model on overlapping material. The person described was closely involved in designing the study and is not an independent participant; to protect their privacy, their relationship to the research is not described further. The method was re-implemented from the paper’s description, and statements were judged one by one by the user rather than scored field by field against a reference. The user checked a selected set, not every statement, and judged statements as true of them, not whether the cited messages support them. No raw transcript excerpts or direct identifiers of other people are included; the examples summarise selected errors while withholding the underlying statements and citations.
How this was done (details)
- Sources. Shorter window: 2,000 consecutive log entries, 1 August–3 September 2026. Whole log: 9,487 entries, 30 May–12 October 2026. Only the user’s own messages count as evidence. The runs excluded messages that were not the user’s in substance: companion summaries the user pasted in for processing, text pasted from other people or AI systems, two messages typed by another agent in the user’s channel, one deliberate test message, and jokes or fiction. Those exclusions were applied to the evidence, and claims left without evidence were dropped.
- Runs. Three runs per reading, Claude Opus 5.5 agents, each blind to the others. Whole-log runs read three time chunks (30 May–4 July, 5 July–10 August, 11 August–12 October); a merger for each run combined its chunks, keeping earlier states of facts that later changed.
- Merging across runs. Statements were clustered by meaning across the three runs (by agents, with their grouping decisions saved), and each merged statement records how many runs found it. Merging can change the proposition being judged; compound statements were judged as worded.
- Selection (decided before checking). After the exclusions above, and after dropping persona paragraphs that only restated profile facts: all statements found by all three runs; all flagged statements (claims resting on something the companion said and the user affirmed, contradictions between runs, Big Five ratings, and health or core-identity statements where runs differed); seeded random samples of 60 one-run and 40 two-run statements (restating persona paragraphs were kept only if flagged). Shorter window: 813 merged statements, 399 checked (seed 2029). Whole log: 1,779 merged, 345 judged by agents already covered by statements confirmed in the shorter reading, 564 checked of the remaining 1,434 (54 flagged; seed 2030).
- Judging. By the user, one statement at a time, with the cited messages available; “true” meant true of them, then or now as worded; the “true then, not now” option existed only for the whole log.
- Agreement. Field-overlap (Dice) between pairs of runs, adapted from the paper’s description; it uses 16 demographic fields, 16 prose sections and 13 identity slots where the paper scores 13, 15 and 7, and it omits persona personality dimensions, which the runs did not produce.