Personality Prediction

What it is

The founding study is Michal Kosinski, David Stillwell and Thore Graepel's work on 58,000 volunteers who supplied Likes, demographic profiles and psychometric test results. Correlations between predicted and actual scores run at about 0.43 for openness and about 0.40 for extraversion, with the remaining traits between about 0.17 and 0.30, and the paper's own benchmark, routinely dropped when the result travels, is that only openness approaches the test-retest reliability of the questionnaire while the rest correspond to roughly half of it. Age and network density were predicted far better than any trait. Wu Youyou and colleagues later reported computer predictions from Likes outperforming judgements by the participants' own Facebook friends.

In effect

Raphael Perla and colleagues' meta-analysis is the correction, and it is the first synthesis this question has had. Searching four databases, they found that only forty per cent of studies could be positively identified as free from data leakage while forty-two and a half per cent were confirmed to suffer from it, and that leakage nearly doubles the reported result. Pooling the leakage-vetted studies gives about 0.23 against about 0.35 for all eligible studies. Their mechanical explanation is that the median study had only six observations per feature and that almost sixty per cent failed to meet even minimal data requirements for linear regression, which makes rigour rather than model class the main determinant of reported accuracy.

What it does not say

It does not say a model can tell you who somebody is. The criterion in every study is a questionnaire score, so the reliability of the self-report instrument is the ceiling on the whole literature, and all three founding papers say so. It does not say the five traits are predicted comparably, since only two of them are predicted reasonably well and those are the two the targeting experiments chose, for exactly that reason. And the headline numbers that circulate are not comparable with each other, because classification accuracies for categorical attributes and correlations for continuous traits are different quantities, and reporting an eighty-eight per cent and a 0.43 in the same breath is the move that makes this field sound stronger than it is.

---