Are Schools Measuring the Right Things? The Problem with 'Usage Metrics' in EdTech

Most New Zealand schools are looking at the wrong dashboards.

Logins, daily active users, streaks, completion rates — these are EdTech metrics, but they are rarely learning metrics. And the gap between the two is where most of the disappointment lives.

What schools think is happening vs what is actually happening

What schools think is happening: The platform tells us how often students log in, how many activities they complete, and how active each cohort is. That data should tell us whether the platform is working.

What is actually happening: Those metrics describe activity. They don’t describe learning. A student can log in daily, complete activities consistently, and still be learning very little. Another student can log in twice a week and learn deeply. The dashboard cannot tell those two students apart.

Worse, the metrics that look healthiest are usually the ones designed by the platform to look healthy. They optimise for engagement signals, not for outcome signals.

Why this keeps happening

There are structural reasons schools end up measuring activity instead of impact.

• Activity metrics are easy. They come pre-built into every platform dashboard. Outcome metrics need to be designed.
• Vendors report on what flatters their product. A platform will surface logins, badges, streaks and completion before it surfaces “students who improved on this topic by two grade boundaries.”
• School reporting cycles need quick numbers. Boards and senior leaders want updates. Logins are available. Outcomes take a term.
• Outcome measurement requires baseline data. If the school didn’t capture a clean pre-implementation baseline, it can’t run a real before/after on academic impact.
• Nobody owns measurement design. The platform vendor owns the dashboard, IT owns the data export, and academic leadership owns the outcomes — but nobody is responsible for designing the measurement framework that connects them.

This is how schools end up reporting confidently on metrics that don’t really answer the question being asked.

The consequences

Measuring the wrong things doesn’t just produce noise. It distorts decisions.

• Renewal decisions are made on engagement rather than learning. The school keeps tools that look busy and drops tools that look quiet, regardless of actual academic value.
• Teachers feel the metrics don’t reflect what they see in class — and lose trust in the platform’s reporting.
• Students who look “active” mask students who look “engaged.” Genuine learners blend into the average.
• Senior leadership ends up defending dashboard numbers that don’t translate into outcomes — which weakens internal credibility for EdTech investment.
• The next platform decision is made under pressure to show “better numbers,” which usually means even more activity-flavoured metrics.

The school is busy measuring. It just isn’t necessarily measuring anything that matters.

What actually works

Schools that get this right tend to build their own measurement spine, with three layers.

1. Hygiene metrics. Are people using the platform at all? Logins, completion, weekly active users. Useful as a heartbeat, not as a verdict.
2. Behaviour metrics. Is the use aligned with how learning works? Time-on-task per session (depth, not surface), spacing of practice (regular vs cramming), feedback engagement (whether students actually open and act on the comments).
3. Outcome metrics. Is teaching and learning visibly different? Topic-level improvement against a baseline, intervention triggered earlier, marking time saved per teacher per week, predicted-grade reliability against eventual results.

The hygiene metrics tell you the platform is alive. The outcome metrics tell you it is doing something. The behaviour metrics tell you why — and let you fix it before outcomes drift.

A note from working with schools

In our work with schools, the conversation about measurement almost always reveals more than the conversation about platforms. When a school can clearly articulate what it expects to change in teaching and learning — and can describe how it will know — implementation tends to succeed regardless of the tool. When measurement is left to the vendor’s default dashboard, even the best tools tend to underperform.

The discipline isn’t analytical. It is editorial — deciding what to ignore, so the things that matter become visible.

If this sounds familiar…

If your school is reporting confidently on EdTech usage but quietly unsure whether the platform is actually changing outcomes, it may be worth stepping back to evaluate the measurement framework, not the dashboard.

We regularly work with New Zealand schools to:

• Audit the metrics currently being reported and identify the gaps.
• Define hygiene, behaviour and outcome layers for each platform in use.
• Build a simple, owned measurement spine that doesn’t depend on vendor reporting alone.

This is not about replacing the platform. It is about making sure the school is measuring the right things to make confident decisions.

A short conversation

If this is something you’d like to explore further, we’d be happy to have a short consultation to understand what your current dashboards are showing — and what they might be missing.

From there, if there is alignment, we can also explore how platforms like AI Buddy fit alongside the right academic and operational support — including the kind of measurement that connects activity to actual learning.