Version 8/15/19
M88-560-01 Survey Methods: Design and Evaluation
Fall 2019
Course Instructor:
Amy McQueen, PhD
Associate Professor, Division of General Medical Sciences
Email: [email protected] (Preferred contact mode)
Office hours (in person or by phone) by appointment
Coursemaster Biography:
Dr. McQueen has taught Survey Methods for AHBR since 2014. She has experience developing and validating new
measures in grant-funded studies. She has many years of experience developing and administering surveys to diverse
research participants in a variety of settings, as well as analyzing both qualitative and quantitative survey data. Her
content area of expertise is cancer prevention and control, specifically the design, implementation and evaluation of
behavioral interventions to increase recommended cancer screening tests and smoking cessation. She also has experience
studying populations with chronic diseases such as type 2 diabetes.
Faculty web page: https://generalmedicalsciences.wustl.edu/directory/amy-mcqueen-phd/
Link to Publications:
http://www.ncbi.nlm.nih.gov/sites/myncbi/amy.mcqueen.1/bibliography/44199745/public/?sort=date&direction=descending
Teaching Assistant:
Tara Ruggeri, MPH
Clinical Research Study Assistant, Division of General Medical Sciences
Email: rugg[email protected]
Welcome:
I get fired up in the classroom to teach students everything I know about survey methods including evaluating the
reliability and validity of survey measures. This can mean I talk too much so I continue to look for ways to incorporate
activities and discussion into my classes so students can get to know each other, learn from each other, and apply what
they read to “real-world” assignments. Everything I ask students to do reflect real decisions, activities, and skills needed
to conduct high-quality survey research; thus, this is a perfect class for students already engaged in related research or
developing a project of their own. I love when students can connect what we discuss in class to real-life experiences or
observations related to survey research so I ask each one to do that in a brief class presentation. This is a challenging
course but students will always get examples and/or practice in class for each writing assignment.
About This Course
Course Description:
This course will provide a comprehensive introduction to survey methods and psychometrics. It will cover theory as well
as practical application. Class time and course assignments will involve many practical hands-on experiences in
designing and evaluating survey measures, culminating in a proposal for validating a new scale of the student’s own
creation. Each student will develop a survey incorporating standard and new measures, which will be administered to
others to practice recruitment, data collection, data entry, and codebook design.
Recommended prerequisites: statistics (linear regression) and research methods
Course Materials:
Students will be expected to have purchased the SPSS grad pack software or have access to version 22 or higher during
the course. Students will not be asked to buy a textbook for this course. Readings from book chapters and articles will be
assigned to supplement the lecture each week. Electronic copies will be shared via our class folder in Canvas. For
students interested in owning useful reference textbooks, the following books are recommended:
Nunnally JC, Bernstein IH. Psychometric Theory. New York: McGraw-Hill 1994.
2
Aday LA, Cornelius LJ. Designing and conducting health surveys: A comprehensive guide. San Francisco, CA:
Jossey-Bass 2006.
Groves RM, Fowler FJ, Couper MP, et al. Survey Methodology. Hoboken, NJ: Wiley 2009.
Dillman DA, Smyth JD, Christian LM. Internet, mail, and mixed-mode surveys: Tailored design method.
Hoboken, NJ: John Wiley & Sons, Inc. 2009.
Technology Requirements:
SPSS Software will be needed to complete at least one assignment. SPSS Licenses can be purchased for 6 or 12
months online from various retailers or from the same site Wash U staff can buy other software at a discount:
https://estore.onthehub.com/WebStore/ProductsByMajorVersionList.aspx?cmi_mnuMain=2ff73789-74c7-e011-ae14-
f04da23e67f6&pc=ddc848d8-b4fe-e111-bd05-f04da23e67f6
SPSS may also be available for temporary use via a trial license through ibm.com. Computers with SPSS
software are also available for student use in the Becker Library & Clinical Research Training Center (CRTC)
on the medical campus.
Time Requirements:
For face-to-face courses it is expected that you will be in class approximately 1 hour per week for each credit of the
course a week plus travel time (i.e. this is a 3 credit course so that is 3 hours a week). In addition it is assumed you may be
doing homework and reading assignments that take at least double that time. You should anticipate your time commitment
for this course to be at least 9 hours a week.
Learning Objectives:
Upon course completion, students will be able to:
Recognize the complexity of survey research and measurement evaluation
Identify and evaluate existing survey measures
Describe factors involved in the organization and length of survey measures
Select and justify the most appropriate survey mode, and develop procedures for survey administration
Recognize the various sources that may contribute to total survey error and methods for reducing them
Identify reasons and methods for creating and evaluating new survey measures
Recognize the different reasons for, and potential problems with, using various response scale options
Propose studies to evaluate the reliability and validity of a measure
Create codebooks for variables created from quantitative survey measures and qualitative data coding
Grading:
5% Class participation; 30% Graded Exercises; 65% Written Assignments
Class participation/Attendance. Students are responsible for reading prior to
class and asking questions as needed in person or by email. Students are expected
to contribute to in-class exercises and discussions including group work. The
instructor should be notified of all absences in advance when possible. Students
who miss more than 3 lectures may receive a deduction in their final grade.
Individual assignments may benefit from group interaction, but must reflect the
student’s own mastery of the material. Thus, individuals in a group may receive different grades for their written work.
Grades/sub-grades
4-point scale
A+ (98% to 100%)
A (93% to 97%)
A- (90% to 92%)
4.00
4.00
3.7
B+ (88% to 89%)
B (83% to 87%)
B- (80% to 82%)
3.3
3.00
2.7
C+ (77% to 79%)
C (73% to 77%)
C- (70% to 72%)
2.3
2.00
1.7
3
Class Schedule
Week
Topic
Description
Assignment
Reading for class
1
8/26
Introduction
Overview of class, field, conceptual
models
Start looking for
existing surveys or
scales (post to folder)
2
9/2
Labor Day Holiday – No Class
3
9/9
Latent constructs
Examples of latent constructs &
multi-item measures, measurement
articles
Discuss distributions/stats
a. Written definition &
operationalization of a
latent construct
(Due 9/17)
4
9/16
Item Response Theory,
classical test theory,
generalizability theory,
factor analysis
Introduce measurement theories and
selected analytic methods; computer
adaptive testing (PROMIS)
Start looking for an
article evaluating a
latent construct of
interest for your survey
Costello article
5
9/23
Measurement reliability
Subjective measures
Reliability (models, methods,
measures)
1. Written evaluation of
reliability and validity
of a published scale
(student’s choice)
(Due 10/11)
Aday Reliability p.54-62
Kimberlin article
6
9/30
Measurement validity
Validity (theory, methods, influences,
measures)
Cross-cultural equivalence
Kimberlin article
7
10/7
Measurement reliability
and validity discussion
Solidify understanding of reliability
and validity
8
10/14
Identify, Evaluate and
Select existing items for a
survey (standard items and
scales); Scales of
measurement
Non-scales; national surveys;
databases of measures
Fixed vs open ended; Response
options; Reverse scored items
2. Create/format
survey of 5+ standard
measures of participant
characteristics and 2+
scales plus describe in
written form.
(Due 11/1)
Read Totten article
before class
Aday Chp 12 Survey
Format
4
Week
Topic
Description
Assignment
Reading for class
9
10/21
Data collection
Survey mode
Mode of administration - pros/cons
across multiple considerations
b. Individual summary
of in-class MODE
assignment in written
form (Due 10/25)
Read Groves Chp 5
Survey Mode before
class
Aday Chp 12 Survey
Format
10
10/28
Methods for adapting
existing measures, pre-
testing & other evaluation
methods
Lit review; Delphi panel; cognitive
interviews; think aloud; observation;
process evaluation
Survey/Scale pre-testing methods
Add new scale to
survey (HW#2)
Moorman article-skim
before class
Blair Chp 10 Pretesting
Optional:
Blair Appendix 3;
Smith article-focus on
first 3 pages; Collins &
Presser articles
11
11/4
Creating new measures.
Qualitative methods
Using qualitative data (historical data,
interviews, focus groups) to generate
content or specify wording
c. Create sample
interview questions,
coding protocol,
codebook (Due 11/12)
Sofaer article
12
11/11
Online vs. Field
procedures
Interviews/Interviewers
Field administration; interview
dynamics; quality control methods
d. Administer survey to
others; Write reflection
on interviewing
(Due 11/19)
Aday Chp 13
Implementation
13
11/18
Data management and
quantitative analysis
Aggregate measures, scoring, dealing
with missing data, implications for
analyses
Creating codebooks
3. Create SPSS dataset
and codebook for full
survey (Due 11/26)
Aday Chp 14
14
11/25
Thanksgiving Week – No Class
(Optional consultation meetings for Final assignment)
15
12/2
Review + remaining topics
Observation/Abstraction
forms
TBD – choose your own adventure
Optional
Aday Chp 6 Sampling;
Groves Chp 6 Non-
response & Chp 11
Ethics
12/9
Exam week - No class
Final assignment due today
4. Final assignment -
Describe new scale and
evaluation plan
Italicized assignments reflect graded exercises or class participation.
5
Assignments. The goal of these assignments is to allow students some personalization of their experience with several key
tasks in survey development and evaluation. A separate detailed description will be provided for each assignment.
Writing Assignments (100 points each)
Due Date (by 5pm!)
Assignment 1: Written critique of a validated multi-item measure of latent construct
10/11
Assignment 2: Turn in formatted survey and describe in written text
11/1
Assignment 3: Turn in labeled SPSS database including data entered for <10
participants, and codebook with scoring instructions
11/26
Assignment 4: Propose a new scale and a validation study
12/9 last class
Assignments are designed to build upon each other. For Assignment 1, each student will evaluate the reliability and
validity of a standard measure of a latent construct from the literature. For Assignment 2, students will each select and
format at least 5 standard sociodemographic and other background measures from public surveys used to describe the
sample, 2 multi-item scales from the published literature of latent constructs, and 1 additional multi-item scale we will
adapt/create in class. Students will also write a description of each measure such as what is included in the methods
section of an article. For Assignment 3, students will administer their formatted survey to a convenience sample of at
least 5 people then enter the survey data they collect into a SPSS dataset and create a codebook including scoring
information for each variable in the dataset. For Assignment 4, students will propose a new multi-item scale of their own
definition and design, and propose methods for evaluating its reliability and validity.
Graded Exercises. The goal of these assignments is to allow students to practice skills learned in class and to
(independently) demonstrate what they learned from group activities.
Graded Exercises (25 points each)
Due Date (by 5pm)
A. List, define, and operationalize a latent construct
9/13
B. Written summary of group presentation on survey mode
10/25
C. Qualitative interview guide, protocol, & codebook for formative research
11/12
D. Reflection on keys to successful survey administration
11/19
E. Individual oral presentation on real-world survey methods example
TBD
Assignments and Exercises should be typed and submitted as word documents. Files should be saved with the student’s
last name and brief assignment name (e.g., Smith Survey Mode Exercise A). Homework files must be uploaded to canvas
by 5:00 pm on the due date. Word documents are preferred to pdf files so that Dr. McQueen can insert comments in the
document using track changes to provide feedback more easily.
Deadlines. There are no alternate assignments that will be given in place of the required assignments described above.
Penalties for late work include 10 points deducted from the final grade per day late. Students are advised to communicate
with the instructor before an assignment is due to plead their case for an extension or temporary “incomplete” grade.
This syllabus is subject to change at the discretion of the professor to accommodate instructional and or student needs.
It is the student's responsibility to keep abreast of such changes.
Individual presentation (~5 min). Each student will present a real-world survey methods example to the class for
discussion. Students should solicit approval and suggestions from the instructor for their selected topic in advance of their
scheduled presentation date. Students are encouraged to draw on their own personal experiences or exposure to research.
Canvas
If you have any technical problems accessing MyCanvas.wustl.edu please e-mail ahbr@email.wustl.edu. Note this
mailbox is not monitored in the evening or on weekends. If you need immediate help after hours please submit a service
request to https://wusm.service-now.com.
6
AHBR Core Competencies for Master’s Degree Candidates
Core Thematic Areas
Competencies
1) Describe the role and interaction of
key determinants of health status,
health behavior, and health behavior
change from a biopsychosocial
perspective across the lifespan.
Describe the impact of socio-ecological, biological, and psychosocial
determinants on health status, health behavior, and health behavior change.
2) Describe and apply relevant
theories, concepts, and models
from social and behavioral science
to both understand and modify
health behavior and health
outcomes.
Identify theories, concepts and models from a range of social and
behavioral disciplines that are used in health behavior research.
Understand how to operationalize and measure theoretical constructs.
Describe some of the challenges of using social and behavioral theories and
models to inform programs involving multiple levels of change (e.g.,
individual, family, community, organizations).
3) Describe and apply basic principles
of research and evaluation
methodology relevant to
understanding and modifying health
behaviors and health outcomes.
Describe qualitative research methods (e.g., focus groups, key informant
interviews) used to understand health behaviors and health outcomes, and to
design and evaluate intervention programs, including appropriate data
collection and analysis techniques.
Apply quantitative methods, e.g., behavioral surveys and biometrics, to
understand health behaviors and health outcomes, and to design and
evaluate intervention programs, including appropriate data collection and
analysis techniques.
Describe the basic principles of study design including interpretation of
alternative hypotheses and threats to internal and external validity.
Critique and synthesize scientific evidence, including evidence review.
Deliver effective scientific presentations.
Describe the basic principles of random variation, systematic error,
sampling error, measurement error, hypothesis testing, type I and II errors,
and confidence limits.
Understand the statistical assumptions behind different statistical methods.
4) Describe and apply ethical
principles relevant to health
behavior research and practice.
Describe the ethical codes and principles relevant in health behavior
research.
Describe rules and regulations designed to uphold ethical principles (e.g.,
HIPAA to protect confidentiality, research ethics requirements to protect
human subjects).
Understand the ethical implications of self-report measures
5) Plan, implement, and evaluate
health behavior programs across
diverse settings and populations.
Apply evidence-based approaches to the development, implementation, and
evaluation of health programs.
Understand the importance of incorporating multidisciplinary perspectives
in program design and evaluation.
6) Describe and apply the knowledge
and skills necessary to interact
with diverse individuals and
communities.
Understand the principles of social justice and equity when interacting with
diverse individuals and community settings.
Understand the concepts and skills involved in designing culturally
appropriate measures.
7) Manage health promotion and
behavior change programs.
Develop, implement, and maintain quality assurance and control procedures
for data collection, data management, statistical analysis and data
monitoring.
Prepare and submit IRB application(s) including all the necessary
documents and study materials.
Develop and administer high-quality data collection tools
Develop and manage qualitative and/or quantitative data including
codebooks and protocols for data management and analysis
7
University Policies and Resources:
Please refer to the AHBR Scholar Handbook for complete program policies and information.
Plagiarism reflects unprofessional and unethical behavior and such use will result in an immediate failure for the course.
Plagiarism, as defined for this course, results if you take 5 or more consecutive words from a written source (including
web pages) without proper citation. You must place the phrase in quotation marks and cite the author, year of publication,
and page (or web publication). Keep in mind, rewording or rephrasing someone else’s writing or ideas should also
include a reference. Not only is proper referencing ethical, but a regular part of our professional practice.
Drop Dates. If the occasion should arise that you want or need to drop this class, please talk to the course instructor first.
You can drop for any reason during the course of the semester, however you my only receive a partial or no tuition
reimbursement after mid-September or depending upon how far into the semester you drop the course. Late withdrawals
will appear on your transcript as a withdrawal. See the university academic calendar for specific dates.
Writing Assistance:
For additional help on your writing, consult the expert staff of the Writing Center in Olin Library on the Danforth campus
(first floor): https://writingcenter.wustl.edu
Disability Resources:
Washington University is committed to providing accommodations and/or services to students with documented
disabilities. Washington University’s Center for Advanced Learning Disability Resources is the University’s official
resource for students with disabilities and students with suspected disabilities. DR assists students with disabilities by
providing guidance and accommodations to ensure equal access to our campus, both physically and academically. To
learn more about its services, initiate the process of formal documentation and/or to arrange for accommodations, please
contact at the start of the course: http://cornerstone.wustl.edu/disability-resources/
Mental Health Resources:
Mental Health Services’ professional staff members work with students to resolve personal and interpersonal difficulties,
many of which can affect the academic experience. These include conflicts with or worry about friends or family,
concerns about eating or drinking patterns, and feelings of anxiety and depression: http://shs.wustl.edu/MentalHealth.
Reporting Policies:
Please also review the CRTC website for policies regarding sexual assault reporting and reporting concerns about bias,
prejudice or discrimination: https://crtc.wustl.edu/courses/class-list/academic-policies/reporting-policies/
Accommodations based upon sexual assault:
The University is committed to offering reasonable academic accommodations to students who are victims of sexual
assault. Students are eligible for accommodation regardless of whether they seek criminal or disciplinary
action. Depending on the specific nature of the allegation, such measures may include but are not limited to:
implementation of a no-contact order, course/classroom assignment changes, and other academic support services and
accommodations. If you need to request such accommodations, please direct your request to one of following
resources: Dr. Karen Winters, MD, Director of Student Health Service; Dr. Gladys Smith, PhD, Sexual Violence
Prevention Therapist and Licensed Psychologist, or Kim Webb, Director of the Relationship and Sexual Violence
Prevention Center at the Danforth Campus. These are confidential resources; however, requests for accommodations will
be shared with the appropriate University administration and faculty. The University will maintain confidential any
accommodations or protective measures provided to an individual student so long as it does not impair the ability to
provide such measures.
If a student comes to me to discuss or disclose an instance of sexual assault, sex discrimination, sexual harassment, dating
violence, domestic violence or stalking, or if I otherwise observe or become aware of such an allegation, I will keep the
information as private as I can, but as a faculty member of Washington University School of Medicine, I am required to
immediately report it to an Office of Education Dean or directly to Ms. Jessica Kennedy, the University’s Title IX
Director. If you would like to speak directly with Ms. Kennedy, she can be reached at (314) 935-3118, or by visiting the
Title IX office in Umrath Hall on Danforth Campus. Additionally, you can report incidents or complaints to the Office of
Student Affairs or by contacting WUSM Protective Services 314-362-4357 or your local law enforcement agency.
8
You can also speak confidentially and learn more about available resources by contacting Dr. Gladys Smith, PhD, Sexual
Violence Prevention Therapist and Licensed Psychologist at the Medical Campus, (314) 362-2404, or contacting the
Relationship and Sexual Violence Prevention Center located on the 4th floor of Seigle Hall on Danforth Campus, (314)
935-3445. For further Resources, see: wusmhealth.wustl.edu/students/victims-sexual-assault-abuse/resources-2/
Bias Reporting:
The University has a process through which students, faculty, staff and community members who have experienced or
witnessed incidents of bias, prejudice or discrimination against a student can report their experiences to the University’s
Bias Report and Support System (BRSS) team. See: diversityinclusion.wustl.edu/brss/. To report mistreatment or
offensive behavior in the MD program, please report via the following pathways:
CANVAS (the Learning Management System) utilizing the direct link from the Student Commons <
Mistreatment, Offensive Behavior, and Bias Reporting
Directly contacting any of the following individuals:
o Senior Associate Dean for Education
o Associate Dean for Student Affairs
o Associate Dean for Medical Student Education
o Student Ombudsperson (as another confidential resource)
On the Medical Campus:
Office of the Associate Vice Chancellor for Diversity, Equity and Inclusion (DEI)
The DEI Training Team designs, facilitates and leads diversity education programming for faculty, staff and students on
a wide range of topics including: creating a climate of respect, the value of diversity and the role of biases in our day-to-
day lives.
diversity.med.wustl.edu/training/
The Office of Diversity Programs promotes diversity among and prepares medical students to lead in a global society. A
priority for the Office of Diversity Programs is to cultivate and foster a supportive campus climate for students of all
backgrounds, cultures and identities.
mddiversity.wustl.edu/
The Diversity and Inclusion Student Council promotes an inclusive campus environment for all School of Medicine
students.
sites.wustl.edu/disc/
The Office for International Students and Scholars embraces the university’s mission of welcoming promising
students from around the world.
wumma.wustl.edu/