Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological review, 84(3), 231.
Though a bit of a difficult read, this is a really great paper, especially with regards to methods.
Summary
This should be called “Why Subjects Will Give You Bad Data Without Knowing It.” Though written with verbal reports in mind, the paper seems applicable to any type of self-report. We perceive ourselves as trait-stable individuals, and regularly discount the impact of context and situation (see Mischel 1973). Additionally, some processing (like perception) appears to take place outside of consciousness, including, as the author argues, some cognition. A whole range of experimental evidence is cited, which gives rise to several conclusions. Often, people are not aware of a stimulus that is affecting their response or behavior. Sometimes, they are not even aware of their response at all. If aware of the stimulus, they are often not aware that it has affected them (sometimes they even insist that it hasn’t). In a similar vein, creative cognitive acts are often unattributable (e.g., “an idea struck me out of the blue”). When people attempt to make a causal attribution, it is generally based on a priori theories (i.e., “X produces Y, and since Y happened, X must have happened as well”) or simply plausibility (the authors cites the representativeness heuristic).
Of special interest to me is the section entitled “When Will We Be Wrong In Our Verbal Reports?” It provides several reasons that can form a sort of checklist for sound experimental procedures. For example, temporal distance is more likely to result in inaccurate self-reports. Instead of asking a subject to “Think back to a time when…”, we would obtain more accurate results by asking immediately after the stimulus. What the authors call mechanics of judgement can also play a role. For example, the order of questions in a survey might unintentionally cause order effects – random ordering of survey items would be preferable. As mentioned previously, we are particularly bad at recognizing contextual factors (e.g., “I’m angry because you’re being a jerk” when you’re actually hungry). The next two deal with salience - nonevents and nonverbal behavior. Nonevents, which did not happen, are generally less salient (and therefore less available for recall) than events which did happen. Similarly, nonverbal behavior is less salient than verbal cues. Finally, we tend to associate large effects with large causes, when many times a small thing can cause a large effect.
Application
This paper has served as a sort of confirmation of a gut-feeling that I’ve had for awhile regarding the primacy of self-report in current psychological research – namely that we rely on it a lot, maybe too much. Evaluation through multiple forms (perhaps a self-report and a behavioral observation) should result in better validity.
Comments powered by Disqus.