Reference bias: what does "good" look like?

A recent study by Heckman & Kautz ranked nations according to their self-reported conscientiousness. They then cross referenced the average number of hours a worker in that country actually worked each year (see below).

National Rank in Conscientiousness vs. Average Annual Hours Worked

Overall there is little correlation between how conscientious people in each country think they are, and the reality of how much they work. As the authors state: “The results are surprising, South Korea ranks second to last in terms of conscientiousness but first in the number of hours worked.” France on the other hand put in 40% fewer hours each year than the South Koreans, yet rate themselves as far more conscientious. Thanks to the Stirling Behavioural Science Blog for sharing the paper.

It is a reminder that measuring abstract concepts in surveys can be a challenge. If participants have to interpret what a concept like ‘conscientious’, ‘happy’ or ‘lazy’ means, they will all reference different norms. This “reference bias” makes comparisons misleading.

What’s needed is common frame of reference. Academics use “vignettes” (short written descriptions) to help anchor responses and give more standardisation across cultures. there is a great list of these here for anyone interested. “Ken” for example, is the anchor point of 5/5 happiness on a happiness scale – and sounds like great company:

“Ken loves life and is happy all the time. He never worries or gets upset about anything and deals with things as they come.”

Whilst commercial constraints are going to prevent the average market researcher developing vignettes for every survey, when you’re measuring abstract notions they reduce subjectivity. I’m resolved to giving them a go.