What Happens When the Person Checks the File (RealCompanion)

21 October 2026

Diagram: a long conversation history on the left is read into a file of statements about the user on the right; beside each statement sits the person's judgement — most ticked as true, one marked partly true, one crossed out as false.

In October 2026, researchers at Quis Lab published RealCompanion, a benchmark built from real conversations between ten people and an AI companion application (Behnam, Kim, Yu, Huang and Yang, arXiv preprint, 2026). It is careful work that documents important limitations, and it asks a question I care about: whether an AI system can understand a person from a long history of conversation with them.

One of its tracks asks an AI system to read a person’s whole history and write a file about them: a profile of things the person stated, each backed by the messages that establish it, and a persona of interpretations, such as personality traits, quirks and working hypotheses about why the person acts as they do. The system’s file is scored field by field against a reference file. The paper summarises this track memorably: the systems “see the person, and then imagine more”, adding about seventy to eighty further fields for every hundred they recover.

That scoring has a natural limit, which the authors acknowledge. Their reference files were built by a pipeline and reviewed by the authors, and the paper states that its trait estimates and clinical-style summaries were not reviewed by the people they describe. So an additional field the system writes can count against its precision without anyone having judged whether it is false. This evaluation did not include asking the person.

This note reports one case in which the person described checked the reconstructed statements.

What I did

The study used one person’s conversation history with a long-running research system I built, which has its own persona, a layered memory and months of shared history. With the consent of the person described, I adapted the reconstruction task to that log. I could not locate the reconstruction prompts in the public repository when I checked it (21 October 2026), so I re-implemented the method from the paper’s description, using its file structure, field names and evidence rules: every profile statement must cite the user’s own messages, and something the system said counts only if the user confirmed it.

Three runs of one model, Claude Opus 5.5 (one of the three systems the paper tests), each read the history and wrote the user’s profile and persona, separately and blind to one another. I did this twice: once on a shorter window (2,000 log entries from all speakers, 1 August to 3 September 2026), and once on the whole log (9,487 entries, 30 May to 12 October 2026), read in three time chunks and merged within each run.

The three runs’ statements were merged, and the user checked a deliberately selected set, one statement at a time, with the cited messages one click away: true, partly true, false or unsure, and, for the whole log, “true then, not now”. After the exclusions and the removal of redundant persona paraphrases described at the end, they checked every eligible statement all three runs had found, every statement flagged as contested or sensitive, and random samples of the statements only one or two runs had found. For the whole log, before any new statements were selected, agents set aside 345 statements they judged already covered by ones the user had confirmed in the shorter reading. The percentages below describe the statements the user checked, not all the statements the runs generated. The details are at the end.

1. Most of the checked statements were true by the user’s judgement

For the shorter window, the user checked 399 statements: 392 were true, 4 partly true, 2 false and 1 they were unsure about. For the whole log, they checked 564 more: 542 were true, 3 had been true but no longer were, 12 were partly true, 6 were false and 1 was unsure. In the random samples of statements that only one of the three runs had found, 56 of 60 were true for the shorter window and 58 of 60 for the whole log.

Stacked bar chart of the user’s verdicts. Shorter window, 399 statements checked: 392 true, 4 partly true, 2 false, 1 unsure. Whole log, 564 further statements checked: 542 true, 12 partly true, 6 false, 3 true then but not now, 1 unsure. Below, random samples of statements only one of the three runs found: 56 of 60 true in the shorter-window sample and 58 of 60 true in the whole-log sample.

Most of the partly true statements were, in the user’s reading, about timing and nuance: a plan whose start date was still open, something they do but not with the person named, a project described as completed when it was still in progress.

So low agreement between these runs did not, by itself, mean a statement was false. That is the narrower point this test can make. I did not build a separate reference file and identify which statements it would have omitted, so this does not measure the truth of RealCompanion’s extra fields; and the user’s judgement that a statement is true is not the same as an audit that the messages each run read actually support it. What it adds is the subject’s judgement, which the reported evaluation did not include.

2. Some errors recurred

The user marked two statements false in the shorter reading and six in the longer one. The examples below illustrate the interpretation errors they found; they are not an exhaustive classification, and they are not an independent assessment of the user’s personality.

3. How consistent were the runs?

Using my adaptation of the paper’s field-overlap measure, the three whole-log runs scored 0.94 on profiles and 0.80 on personas. These numbers mix exact-value comparisons with checks that a section is simply present, so they do not measure agreement on every statement; and runs can agree while repeating the same mistake. Ignoring capitalisation in demographic fields (so that, to take an invented illustration, “City” and “city” match) raises the persona score to 0.84; the agreement breakdown also records other wording differences and differences in which quirks the runs found. My field selection differs from the paper’s, so these are descriptive results for this implementation, not a replication of its reported persona run agreement of 0.93–0.95.

4. Earlier messages can change what a claim is supported by

In the shorter window, the runs set aside several statements as “things only the companion said”, which the evidence rule rightly excludes. The whole-log runs reported earlier messages from the user supporting some of those statements. That is consistent with the companion drawing on what the user had told it months before, but it does not establish how the companion produced those statements. The general lesson is modest: a claim unsupported within a short window is not necessarily unsupported in the wider history, and a reference built from less material can omit support available to the reader.

What this means

Reconstruction from a real history is a valuable task, and RealCompanion makes it possible to study. This single case adds what that evaluation did not include: the subject’s own judgement. With it, most of the selected statements, including most of those only one run produced, were true by the user’s account, and the examples of error included effects of health read as personality, a premise offered to the companion read as the user’s belief, and reversals and misreadings. Those are worth measuring directly, because a companion that stores such an inference may act on it for a long time.

One possible explanation for the high rate is that this user corrects the companion early and often: readers of the log inherit a history in which many mistakes were already caught. This study did not test that. A follow-up could compare matched records with and without the corrective messages, and score both the subject’s endorsement and the support in the text, bearing in mind that corrections tend to cluster exactly where errors and difficult topics arise.

I research long-horizon human–AI interaction and built this system’s harness myself (the software that manages its prompts, memory and tools). Its user is not a typical one: they know the system well, were closely involved in this study’s design, and correct the companion often. This note cannot say how a reconstruction would fare for someone who does not. It shows what becomes visible when the person is asked to check the file.

Limitations: one consenting user and one system’s log; three runs of one model on overlapping material. The person described was closely involved in designing the study and is not an independent participant; to protect their privacy, their relationship to the research is not described further. The method was re-implemented from the paper’s description, and statements were judged one by one by the user rather than scored field by field against a reference. The user checked a selected set, not every statement, and judged statements as true of them, not whether the cited messages support them. No raw transcript excerpts or direct identifiers of other people are included; the examples summarise selected errors while withholding the underlying statements and citations.


How this was done (details)