Binary Response Format or 11-Point Scale? Measuring Justice Evaluations of Earnings in the SOEP

Abstract

Questions on justice of earnings are regularly fielded in large-scale surveys but insights intothe role of response formats on measures of the justice of earnings are missing. This problem isillustrated by the German Socio-Economic Panel Study (SOEP), which, in 2017, changed its question onthe justice of one’s own earnings from a binary response scale to an 11-point scale. Meanwhile, the shareof respondents evaluating their earnings as just dropped considerably, leaving unclear howmethodological and substantive effects are intertwined. Addressing this gap, we analysed a surveyexperiment in the 2016 Innovation Sample of the SOEP (SOEP-IS). In a split-ballot design, 2562 employedSOEP-IS respondents were randomly allocated to one of two experimental groups: receiving either thebinary scale or the 11-point scale. Our results show that a lower share of respondents evaluated theirearnings as just in the 11-point scale condition. However, follow-up questions on the just amount ofearnings were unaffected by the question format. We conclude that it is crucial for researchersinvestigating justice evaluations of one’s own earnings to account for these measurement effects as wellas for practitioners to carefully document and test the effects of changes in response format.