A good EdTech pilot answers one question with criteria agreed before it starts. That question might be “Does this platform increase weekly practice in Year 10 chemistry?” Run it for 30 days with one department, measure use, learning and teacher time weekly, and decide on the pre-agreed thresholds. Most pilots fail because they start without a question.
Why do so many EdTech pilots fail?
Because they test the product, not a need. The EEF warns that new technology “can become a solution in search of a problem unless it is introduced in response to an identified need”. It adds that “without a clear plan for support and implementation, technology is much less likely to have an impact” (EEF). A pilot that starts with “let’s see what teachers think” usually ends with mixed views and no decision. See why EdTech pilot programmes fail.
What should a pilot’s question be?
One that links a need to a measure. The EEF’s first recommendation is to “consider how technology will improve teaching and learning before introducing it”. Its other three name where technology can help: “the quality of explanations and modelling”, “the impact of pupil practice”, and “assessment and feedback”. A pilot question should pick one:
| Need | Pilot question | Measure |
|---|---|---|
| Too little practice between lessons | Does the platform increase weekly practice? | Questions attempted per student per week |
| Marking load | Does it reduce teacher marking time without losing information? | Teacher hours on marking, before and during |
| Gaps found too late | Does it show topic gaps before the mock? | Topic gaps identified and retaught |
| Weak feedback | Do students act on feedback? | Improvement on repeat attempts |
What does a 30-day pilot look like?
Our recommended plan:
| Week | Focus | Key actions |
|---|---|---|
| Before day 1 | Set up | Agree the question, success thresholds and decision owner; baseline data; accounts; one department, two or three classes |
| Week 1 | Launch | Teacher training; students start in lessons, not only as homework; check logins and access daily |
| Week 2 | Routine | Set weekly tasks; teachers use the data in at least one lesson; first check-in against measures |
| Week 3 | Depth | Use the tool for its core purpose, such as a practice cycle or mock; collect teacher time data |
| Week 4 | Decide | Final data against thresholds; student and teacher survey; decision meeting |
The EEF’s implementation guide organises change into “manageable phases: Explore, Prepare, Deliver, and Sustain” (EEF). A 30-day pilot is the Explore phase done properly. It should end with a decision to prepare a wider rollout, or to stop.
What should the success criteria be?
Numbers agreed in advance, not impressions afterwards. Set thresholds for three things:
- Use: for example, the share of students completing the weekly task by week 4.
- Learning: for example, improvement on repeat attempts at the same topics.
- Teacher time: for example, marking hours saved per teacher per week.
Add one qualitative check: would the department’s teachers choose to keep using it? Write the thresholds down with the decision owner before day 1. If the tool meets them, the answer is yes; if not, no.
How do you keep students using it past week two?
By building it into lessons and following up. The EEF notes “the relationship between technology, motivation and achievement is complex”. It recommends “monitoring how technology is being used”. Use can drop once novelty fades, so the pilot must test the routine, not the launch. See keeping students on a learning platform after week two.
What should the decision meeting cover?
Four items, in 30 minutes:
- Results against each threshold, from the data.
- Teacher views, from a short survey.
- Cost and scale: what a full rollout would cost, including training time. See EdTech ROI.
- Decision: yes (plan the rollout), no (stop), or extend once with a narrower question.
Also check data protection before any rollout. See student data protection across borders and the EdTech procurement checklist.
How schools do this with AI Buddy
AI Buddy offers a 30-day free pilot with “full platform access” for “up to 50 students”, “weekly progress reports” and “no commitment” (AI Buddy for schools). Schools typically run it with one department around a practice cycle or mock. The weekly reports give the use and learning measures a decision meeting needs. The decision, and the thresholds, stay with the school.
Frequently asked questions
How long should an EdTech pilot run?
Long enough to test a routine, not just a launch: 30 days, covering at least one full practice or assessment cycle, is a practical minimum for a classroom platform.
What are good success criteria for a school pilot?
Thresholds set in advance for use (e.g. weekly completion), learning (e.g. improvement on repeat attempts) and teacher time (e.g. marking hours saved), plus whether teachers would keep using it.
How many classes should be in a pilot?
Enough to see patterns but small enough to support closely: typically one department, two or three classes.
Why do EdTech pilots fail?
Most start without an identified need or success criteria. The EEF warns that technology “can become a solution in search of a problem”.
Discover how AI Buddy helps schools strengthen teaching, learning and evidence-informed school improvement. Or start a short consultation with our schools team using the form below — we will get back to you directly.
Sources
- Education Endowment Foundation, Using Digital Technology to Improve Learning: summary of recommendations (2019)
- Education Endowment Foundation, A School’s Guide to Implementation (2024)
- Tutopiya, AI Buddy for schools: free pilot