Abstract

J.L. Esposito "Some Thoughts on the Use of Field Tests to Evaluate Survey Questionnaires"

This paper focuses on field-based evaluation work that incorporates multiple evaluation methods. Field tests are complex, resource-intensive, collaborative operations that draw upon the knowledge/information/data and skills possessed by various sources/agents (e.g., content and design specialists; interviewers and other field staff; respondents; statisticians) to optimize questionnaire design for the purpose of gathering high-quality data about a particular domain-of-interest. Because field tests represent evaluation work that occurs during specific phases of the questionnaire-design-and-evaluation process, the paper begins with a brief overview of this process. This discussion will be limited to a cursory description of a framework that the author has found useful for situating field tests within the broader context of longitudinal (and potentially reiterative) design-and-evaluation work. With the framework as context, the author attempts to categorize some of the methods and techniques that survey practitioners have at their disposal for evaluating questionnaires at various phases of the design-and-evaluation process. At that point, the author moves from the more abstract discussion of frameworks and methods to a more pragmatic discussion of field-test methodology in real-world settings and does so by reconstructing what transpired in the course of an evaluation of a supplement questionnaire that involved a series of three separate field tests. In the process, the author provides examples of method-generated qualitative and quantitative data, reviews how such data were analyzed and integrated, and offers some thoughts as to the utility of the various methods used. In the final section of the paper, some closing thoughts on the collaborative nature field-test methodology are provided.