Designing Research Surveys: Better Questions, Reliable Measures and Pilot Testing

Translate the research question into measurable concepts

Before opening a survey builder, write down the concepts the study needs to measure. Student engagement, satisfaction and belonging are related but different. A question about enjoyment cannot automatically stand in for participation or learning. Define each concept and explain what observable response would provide useful evidence.

Create a question map linking each item to an objective, planned analysis and intended interpretation. Remove items that are interesting but unnecessary. This reduces respondent burden and limits the collection of information that the project cannot justify. Consider whether administrative data, observation or interviews would answer part of the question more effectively than self-report.

Draft clear and answerable questions

Use language familiar to the target group and specify the period or situation respondents should consider. Ask one thing at a time. A question about helpful and timely feedback combines two judgements; a respondent may find feedback helpful but late. Separate these dimensions if both matter to the study.

Make response categories understandable, appropriately ordered and sufficiently complete. Distinguish not applicable, do not know and prefer not to answer when those meanings differ. AAPOR's guidance supports concise wording, attention to question order and testing with respondents similar to the intended sample. ^[1] Treat these principles as a starting point for deliberate item design rather than a mechanical checklist.

Evaluate measures before using them

When an existing scale appears suitable, check its purpose, population, language, permissions and evidence of measurement quality. A familiar scale is not automatically valid in a different setting. Changes to wording, response categories or translation may alter what it measures. Document modifications and assess whether additional testing is needed.

Distinguish reliability from validity. Consistent responses do not by themselves establish that an instrument measures the intended concept. Examine whether items cover the concept adequately, whether respondents understand them as intended and whether interpretations are supported. Obtain specialist advice when the study depends on a new or substantially adapted measure.

Test the respondent experience

Conduct cognitive interviews or other appropriate pretesting to learn how people understand questions, retrieve information and select answers. Ask participants to explain their interpretation in their own words. A polite statement that a questionnaire looks fine is less informative than observing where a respondent hesitates or uses an unexpected meaning.

Test the survey on the devices, languages and assistive technologies relevant to the population. Check branching, required fields, progress indicators and the ability to stop. Sensitive questions should be approached with care and a clear explanation. Ethical guidance for educational research provides an additional basis for considering participant choice, power relationships and potential harm. ^[2]

Illustrative example: evaluating library access

Suppose a university asks, How satisfied are you with the library? Responses may combine opening hours, staff support, digital subscriptions and study space. The result offers little guidance about what to change. The team instead defines separate objectives concerning access, use and service experience.

During pretesting, commuter students explain that they interpreted access as physical opening hours, while distance learners thought of online login problems. The revised questionnaire uses separate, clearly framed items and routes respondents to relevant sections. The pilot also reveals that a required question about borrowing prevents non-users from completing the survey. Fixing that issue improves the instrument before the main fieldwork.

Pilot the complete study process

A pilot should test invitation delivery, consent information, survey completion, export, coding and preliminary analysis. Check whether the resulting dataset can answer the planned questions. Inspect missing values, unusually repetitive patterns, contradictory responses and unexpected completion times without automatically treating every unusual record as invalid.

Set rules for exclusions and partial responses before analysing the main findings. Record any changes made after piloting and decide whether pilot responses can legitimately be combined with the final dataset. If wording or procedures changed substantially, combining them may obscure important differences. Preserve the final instrument and its version alongside the analysis plan.

Report enough for readers to judge the evidence

• Explain the target population and recruitment method.
• Provide the questions and response options where feasible.
• Describe adaptation, translation and testing.
• Report dates, mode, eligibility and usable response numbers.
• Explain missing-data and exclusion decisions.
• State limitations of coverage, self-report and measurement.

A well-designed survey produces interpretable evidence, but it does not remove all uncertainty. Respondents may remember imperfectly or participate selectively. Conclusions should reflect those limits and distinguish expressed perceptions from independently demonstrated behaviour or learning.