Evaluating Educational Technology: Pilot Design, Learning Outcomes and Procurement Decisions

Start with a problem worth solving

A technology proposal should identify an educational or operational problem in terms that can be examined. Perhaps students receive feedback too late, a practical skill is difficult to rehearse, or staff cannot provide enough low-stakes practice. Define the affected learners, existing process and consequences before considering a product.

The Education Endowment Foundation's digital technology guidance examines technology in relation to learning.^[1] For a local decision, write the proposed mechanism: the tool will enable a particular activity, which is expected to support a specified outcome. This makes the claim testable and exposes whether the proposed benefit depends on changes in teaching practice.

Examine the available evidence

Ask suppliers for evidence relevant to the intended learners, subject and setting. Distinguish independent evaluation from promotional case studies and user testimonials. Read the design, comparison, sample and limitations of a study rather than relying on its headline. A positive result in one context does not guarantee the same result elsewhere.

Check whether the evidence concerns the product version being purchased and whether the required implementation conditions are realistic. Identify missing evidence explicitly. The absence of a strong study does not automatically prohibit a small experiment, but it should affect the scale of commitment and the strength of the claims made.

Design a bounded pilot

Select a manageable group and define the duration, support arrangements and exit conditions. Decide what the pilot can answer. A feasibility pilot might assess usability, workflow and staff demand; a more rigorous evaluation may examine learning effects. Do not describe the former as proof of the latter.

Agree success criteria before starting. These may include completion of a learning task, accuracy of feedback, reduction in avoidable administration or learner access. Establish a baseline and an appropriate comparison where possible. Consider consent, privacy and fairness requirements, especially if the pilot changes assessment conditions or collects new personal information.

Collect a balanced evidence set

Measure learning or service outcomes alongside implementation fidelity, staff time, technical problems and user experience. Record training and support costs. Ask which learners struggled to access the tool and whether an alternative route worked. A high adoption rate is not sufficient if the activity contributes little to the learning objective.

Jisc's curriculum design guidance encourages attention to the wider learning experience.^[2] Use that perspective to examine whether the technology improves a coherent teaching sequence or adds a disconnected task. Include evidence from actual student work where appropriate rather than depending entirely on satisfaction surveys.

Illustrative example: automated practice feedback

Imagine a department piloting a tool that gives students feedback on introductory quantitative exercises. The pilot tests whether feedback is accurate, understandable and available when needed. Teachers inspect a sample of responses and record the time required to correct misleading guidance.

Students complete a separate task requiring them to explain the reasoning. If they can answer familiar tool questions but struggle with the explanation, the department investigates whether the activity supports understanding or only pattern matching. It may revise the teaching design before expanding the licence. The pilot does not justify a causal claim unless its evaluation design supports one.

Compare full costs and risks

Estimate the total cost over a consistent period: licences, configuration, integration, content development, staff training, support, accessibility work and exit. Examine data handling, vendor dependence and what happens if the service changes or disappears. Review contractual terms with the appropriate institutional specialists.

State the opportunity cost. Money and staff time allocated to the tool cannot be used simultaneously for other improvements. Compare the proposal with plausible alternatives, such as redesigned feedback practices or additional tutorial support. A technology investment is stronger when it can explain why this option addresses the problem better under the institution's constraints.

Decide, document and revisit

Prepare a decision note summarising the problem, pilot evidence, limitations, costs and recommendation. Possible outcomes include adoption, a revised pilot, limited use or discontinuation. Record conditions of approval and who will review performance after implementation.

Continue monitoring when the tool scales because workload, learner diversity and integration demands may change. Retire features that are not useful and communicate material changes. A responsible procurement decision is not a prediction that technology will transform learning; it is a documented judgement that the available evidence and implementation conditions justify a particular investment.