Home 
Contact 
Profile 
Videos 
Designing Questionnaires and Survey Instruments
Facilitator:
Prof Omar Hasan Kasule Sr.
Workshop Outline
1. Introductions and orientation
2. Background issues in using questionnaires and surveys
3. Practical steps in instrument design and construction
4. Practice in revising items
5. Summary and wrap-up
1. Introductions and Orientation
Workshop Goals
Orient you to conceptual and pragmatic issues in questionnaire and survey design
Describe steps to take in instrument design process
Hands-on practice applying principles
Learn from each other
Identify resources for guiding your future design and implementation efforts
2. Background issues in using questionnaires and surveys
The Social Context
Completing a questionnaire is an act of social exchange
You are getting valuable data
What motivates the respondent?
•Prestige, public recognition
•Money, prizes
•Coercion
•Self-interest, opportunity to influence something
Influencing the Social Exchange
Rewards (anticipated, intangible)
Costs (anticipated, intangible)
Trust (that rewards will outweigh costs)
Establish trust (e.g., authority, token)
Increase rewards (e.g., regard, effectiveness)
Decrease social costs (e.g., inconvenience)
Social exchange vs. economic exchange:
$5 advance token vs. $10 for completion
$5 advance token vs. $10 for completion
Terminology
Questionnaire
Interview
Survey
Survey Methods
Self administered
by mail/internet
in person
by interactive voice response
by mail/internet
in person
by interactive voice response
Interviews
by telephone
in person
by telephone
in person
Survey Methods
Advantages/disadvantages
Advantages/disadvantages
Reliability and Validity
Reliability
•Test-retest reliability (stability)
•Split-halves and Cronbach alpha (internal consistency)
Validity
•Most relevant to external, objective constructs (e.g., events, activities, experiences)
•Less relevant to internal, subjective constructs (e.g., attitudes, opinions, preferences)
How reliable are these items?
How many times in the past year have you seen or talked with a doctor about your health?
Do you favor or oppose managed care?
I would like you to rate different features of your healthcare as very good, good, fair, or poor.
Access
Affordability
Response time
Access
Affordability
Response time
3. Practical steps in instrument design and construction
Section 3 Overview
Exercise
Questionnaire design and construction
Writing items
Response options
Implementation
Section 3 Overview
Exercise
Questionnaire design and construction
Writing items
Response options
Implementation
Steps in Questionnaire Development
Design a blueprint—What am I asking?
Don’t reinvent the wheel—Has anyone else done this?
Constructing questions—How do I ask?
Selecting response format—How do they answer?
Design for data entry—How do I record?
Blueprint
Elements of a blueprint
•List objectives for the questionnaire/survey
•Identify what responses you would like to report
•Determine administrative details
•Review by knowledgeable experts
Don’t Reinvent the Wheel
Literature review
•If you identify something relevant, contact author
Sources for instruments/scales
•Buros Mental Measurement Yearbook
•HAPI
•What else do people recommend???
Questionnaire Outline
1.Introduction
2.Instructions
3.Questions
4.Thank you
Outline: 1. Introduction
Purpose and relevance to them
•Who is supporting (letterhead)
•Personalize address and salutation
•What is requested
•Why selected
•Usefulness of survey
•Appreciation for completing
Outline: 1. Introduction
Operational information
•Anonymity/confidentiality of responses
•When does the survey need to be completed
•How does the respondent return the completed survey
Thank you and “real” signature
Outline: 2. Instructions
Almost nothing is “self-explanatory”
Place instructions where needed
Specify things like
•How to respond when I don’t know
•Circle, check, underline responses
•Do/do not make multiple responses
•“If yes, go to..”
Outline: 3. Questions
Follow a logical pattern
Group similar questions together
Leave sufficient room for open-ended responses
Watch the length of the questionnaire
Outline: 3. Questions
Pay attention to aesthetics
•Don’t make the page too busy
•Make things easy for the respondent
•Use readable typeface and font
Outline: 4. Thank you
Indicate the end of the survey
•Thank you
•Submit
•The end
Repeat instructions to return survey
•Fax number
•Address
•Contact name and phone #/e-mail address
Section 3 Overview
Exercise
Questionnaire design and construction
Writing items
Response options
Implementation
Writing the Items
What are potential sources?
•Literature
•Theory
•Previous surveys
•Focus groups
Writing the Items
Revisit your research question(s)
•What do you want to learn?
•What can your respondents tell you?
•Identify one or more questions that will help answer your question(s)
Writing the Items
List your research variables
•Independent variables
•Dependent variables
•Moderator variables
•Confounding variables
Facts v. Opinions/Attitudes
Facts
•Recall bias
•Reflects behavior
•Predictive of future behavior
•Good indicator of opinion
•Tied to present or past reality
Opinions/Attitudes
•Reflect current thinking on subject
•May or may not correlate with behavior
•Tied to motivation
•No recall bias
•Predictive of future actions
Writing the Items:
Common Problems
Common Problems
Asking questions they can’t answer
Double-barreled questions
Unclear references
Excessive jargon
Implicit negatives
Leading questions
Problem: Questions they can’t answer
How much will funding for your residency program change in the next three years?
How many physicians attending your CME activities reduce office hours in order to attend?
Problem: Double-barreled questions
Should third and fourth year medical students have more options for electives?
How often do your attendees arrive late or leave early from your GME activities?
Problem: Double-barreled questions
Should third and fourth year medical students have more options for electives?
How often do your attendees arrive late or leave early from your GME activities?
Problem: Unclear references
Do you believe that graduating medical students are better prepared for specialty training?
Has attendance at noon report changed in the past year?
Do you believe that new GME models can be built around prevailing theories of adult learning?
Problem: Excessive Jargon
Are you using PDAs in your education programs?
Is your course-tracking software web-based?
Are you currently working with your institutional CQI office to develop training modules in ACGME core competencies?
Problem: Implicit Negatives
Do you support further reductions in working hours for residents?
Do you believe that CME providers should have more restrictions placed on commercial support for educational programming?
Problem: Leading Questions
Do you believe that training in professionalism would be worthwhile for medical students?
Do you support the extensive efforts of the AMA to limit inappropriate gifts to physicians from pharmaceutical companies?
Section 3 Overview
Exercise
Questionnaire design and construction
Writing items
Response options
Implementation
Response Options
Open-ended v. closed-ended
Concrete options for factual items
Attitudes
•Some common Scales
– Likert scale – Ranking
– Semantic differential – Paired comparisons
– Visual analog – Weight allocation
•Unipolar v. bipolar (neutral point?)
Response Options:
Open- v. Closed-ended
Open- v. Closed-ended
Open-ended
•longer to complete
•requires thought
•often left blank
•limited control
•more difficult to analyze
•good follow-up to closed ended questions
Closed-ended
•quick to complete
•focuses responses
•subject to response sets
•easier to analyze
•doesn’t allow explanation of response
Response Options:
Concrete Options
Concrete Options
Factual questions typically have concrete response options.
Information to fill in or check off
•Demographic data (age, gender)
•Employment status (job title, % time)
•Symptom list
•Products owned
Response Options:
Attitudes/Opinions
Attitudes/Opinions
Likert scale
Semantic differential
Effective 1 2 3 4 5 6 7 Ineffective
Visual analog
0 ___________________________________ 100
Response Options:
Common Scales
Common Scales
Ranking
Rank order your specialty preference
(1 = most preferred, 6 = least preferred)
__ Int. Medicine __ Fam. Medicine __ Surgery
__ Pediatrics __ Ob-Gyn __ Psychiatry
Paired comparisons
For each pair, put a checkmark by the specialty you most prefer
__ Internal Medicine vs. Radiology __
__ Radiology vs. Pathology __
__ Family Medicine vs. Radiology __
Response Options:
Common Scales
Common Scales
Weight allocation
Assign 100 points to the following options to reflect your learning preferences.
___ Reading
___ Lecture
___ Small group discussion
___ Individualized instruction
___ Learning projects
___ Other: ______
100 Total
Response Options: Polarity
Unipolar
•none – a lot
•never – always
Bipolar
•strongly disagree – strongly agree
•definitely yes – definitely no
Response Options: Neutrality
Neutral point
•Should you have one?
•Is neutral a legitimate option?
•Will respondents hide their opinions behind neutrality?
•Labeling: “neutral,” “both agree and disagree,” “no opinion”
Which is Better? Why?
Which is Better? Why? Indicate your career goal(s)
| Indicate your career goal(s) No Yes a. Clinical Practice O O b. Teach O O c. Research O O d. Administration O O |
How old are you (to the nearest year)? ___
Or
Check the appropriate age group:
•<18 years
•18-25 years
•>25 years
Or
Please provide your date of birth: __/__/___
An Example
Instructions: All of the questions below are about what you actually do. If you “Always” do what the statement says, circle the 1 for A. If you “Often” do what the statement says, circle the 2 for O. If you “Seldom” do what the statement says, circle the 3 for S. If you “Never” do what the statement says, circle the 4 for N.
1. I think about what I will do 1. A 2. O 3. S 4. N
when I finish medical school.
2. I think about what specialty 1. A 2. O 3. S 4. N
I will practice in someday.
3. I talk to physicians about 1. A 2. O 3. S 4. N
my careers choice.
Section 3 Overview
Exercise
Questionnaire design and construction
Writing items
Response options
Implementation
Implementation: Sample
What is important is response rate
•Better to have a smaller sample from which you receive an adequate response
Be able to describe non-respondents
Number of responses needed
•There is a whole literature on calculating sample size
Implementation: Follow-up
Follow-up to initial mailing is essential to producing convincing results
Include another copy of the questionnaire
Postal card reminder improves response slightly
Most successful follow-up is with phone contact
How high should the response rate be?
As high as possible
80% Excellent
70% Very Good
60% Good
50% Adequate
Survey Development Check List
Self-explanatory and easy to read
Appropriate reading level & language
Proof, proof, and proof again
Pilot test
•Annotate survey for improvements
•Record time
Ready to live with the items forever?
5. Summary and
wrap-up
wrap-up
Summary
Enlisting subjects -- remember sampling issues
Keep it simple & clear
Remain unbiased
Language barriers -- obvious & not so obvious
Timetable -- respondent's & researcher’s
Record keeping
•Keep up-to-date
•Tracking: types of data needed
Resources
Existing instruments
•Buros Center for Testinghttp://www.unl.edu/buros/
•Health and Psychosocial Instruments
http://nihlibrary.nih.gov/
http://nihlibrary.nih.gov/
Resources
For developing surveys
•Harrison RV. Simple questionnaire studies. Journal of Continuing Education in the Health Professions 17(4):228-237, 1997.
•Dillman DA. Mail and internet surveys: The tailored design method. New York: John Wiley & Sons; 2000.
Resources
For developing surveys
•Schuman H, Presser S. Questions and answers in attitude surveys: Experiments on question form, wording, and context. Thousand Oaks, CA: Sage; 1996.
•Fowler FJ, Jr. Improving survey questions: Design and evaluation. Vol 38. Newbury Park, CA: Sage; 1995.
For telephone surveys
•Bourque, L.B. How to conduct telephone surveys (The Survey Kit Series), Vol. 4. Sage Publications, Oct. 2002
Internet Resources
1)Using Interviews in a Research Projecthttp://www.trentfocus.org.uk/Resources/using_interviews_research_project.htm
2) Surveys & Questionnaireshttp://www.trentfocus.org.uk/Resources/surveys_questionnaires.htm
3) Survey design considerations
Internet Resources (continued)
4) Questionnaire design http://www.cc.gatech.edu/classes/cs6751_97_winter/Topics/quest-design/
5) What is a survey?
http://www.whatisasurvey.info/
http://www.whatisasurvey.info/
6) The American Statistical Association:
Survey Research Methods Section
Survey Research Methods Section
Resources
For sample size calculations
•Fink, A. (1995) How to sample in surveys. Thousand Oaks, CA: Sage Publications.
Leftover slides
Types of Interview
Unstructured
Semi-structured
Structured (including questionnaires)
Strengths and limitations of each?
Unstructured Interviews
Strengths
•Qualitative insight
•Hypothesis generation
Weaknesses
•Poor quantitative
•Complex
•Time consuming
Semi-structured Interviews
Strengths
•Qualitative
•Confirm general hypotheses
•Generate specific hypotheses
Weakness
•Limited quantitative
•Complex
•Time consuming
Structured Interviews
Strengths
•Quantitative
•Aggregation
•Less expense
Weaknesses
•Incorrect assumptions
•Qualitative differences
Data Collection Methods
• Personal Interviews
• Telephone Interviews
• Mailed Questionnaires
• Emailed Questionnaires
• Internet Questionnaires
Strengths and limitations of each?
Data Collection Methods
Personal Interviews
+ High response rate, long length, clarification and probing possible, depth of data
- High cost, time intensive, small sample, lower reliability
Telephone Interviews
+ Moderate on response rate, length, clarification opportunities, more efficient that personal interviews, can enter data concurrently
- Moderate cost, respondents must have phone, lower reliability
Data Collection Methods
Mailed Questionnaires
+ Low cost – large volume, efficient, moderate to high reliability
- Lower response rate, limited length, rigidity, limited depth
Emailed Questionnaires
+ Low cost, moderate to high reliability, efficient
- Low response rate, very limited length & format, respondent access and use, limited depth
Internet Questionnaires
+ Customizable, diverse formatting options, immediate data, no incremental cost for large samples, moderate to high reliability, efficient
- Costly development, technology issues (compatibility), respondent access and use, limited depth
Interviewer Considerations
Appearance & demeanor
Language
Must remain unbiased
Familiarity with the questionnaire
Follow instructions & wording exactly
Record responses exactly
Probe for responses appropriately
Phone Survey Considerations
Make an appointment for a phone survey
Don’t sound like a telemarketer!
Be attentive to time; people can hang up
Cannot make observations
Questionnaire Considerations
Length
•Effect on response rate
Incentives
•Token level Improves rate
Appearance
•Color
•Stapled vs booklet
•Professional
What can you learn?
Knowledge
Opinions/Values
Beliefs/Attitudes
Self-reported behavior
Demographic characteristics
Designing Questionnaires and Survey Instruments
10 Nov. 2005 MERC Workshop
Facilitators: O’Sullivan, Gruppen
STRONGLY DIAGREE | DISAGREE | NEUTRAL | AGREE | STRONGLY AGREE |
1 | 2 | 3 | 4 | 5 |
Self-admin Mail | Self-admin In person | Interview Phone | Interview In person | ||
Cost | ++ | + | - | - | |
Time | ++ | + | - | - | |
Standardization | + | + | +/- | + /- | |
Depth/detail | - | - | + | + | |
Response rate | - | ++ | + | ++ | |
Missing Responses | - | ++ | ++ | ++ |