Methodological Effects of Factorial Surveys in Population Samples

Abstract

The factorial survey is a method of attitude research that combines the advantages of survey research and experimental designs. Respondents react to hypothetical descriptions (vignettes) instead of answering single-item questions. By varying the values of each attribute (dimension) systematically the impact of each dimension on respondents' judgments can be estimated. As the descriptions are based on a number of dimensions and respondents have to evaluate between 10 and 20 descriptions factorial surveys are demanding with regard to cognitive ability and the capability of information processing. There is little empirical knowledge if and to what extent this high complexity causes methodological artifacts. The problem becomes apparent when factorial surveys are conducted in population samples where the cognitive abilities of respondents are heterogeneous. Using two sources of information – interviewer evaluations and respondent's behavior – this paper asks for the effects of educational background and age on the comprehension and performance within a factorial survey conducted in a random population sample (N= 1.066) in Germany. Analyses of interviewer evaluations show that: (1) older respondents have indeed to a higher degree comprehension problems than younger respondents, (2) there are almost no differences in response time but in consistency and information (number of attributes) used between educational and age groups, and (3) respondents take a decreasing number of dimensions into account as the number of vignettes increases. This is interpreted as a fatigue effect. All in all, the results demonstrate that factorial surveys are applicable in population surveys but should be used with a smaller number of vignettes per respondent than in student samples.