Space: Dan Meyer On Personalization

Here’s Einstein:

Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.

So what can computers count? The enthusiasm for personalized learning (one of the “trends to watch” in our assignment description) requires we answer that question.

If you’re Netflix, personalization means you know a) what movies a customer has rented and b) how she’s rated those movies on a scale from one to five. That’s what can be counted. The system counts it and then does its best to recommend another movie you’ll like. Netflix does a good job with those two data points, but it’s far from perfect.

Netflix might do a better job personalizing your movie-watching experience by having you type in a brief paragraph explaining your ranking. (Did you like a particular actor? Did one of the underlying themes resonate with you?) Then a Netflix employee would read that information and send you a recommendation.

I’m not saying that’s a good idea. That kind of personalization would be very expensive, for one. But that’s the axiom.

Good personalization is expensive.

Many math education startups (including those that have caught the eye of our instructors) have already met the challenge of cheap personalization, but good personalization still eludes them. Cheap personalization is easy if you look at math from the right angle. From that angle you see a lot of binary, right-or-wrong answers. You see multiple choice questions. You see text fields that are easy to parse as integers and then evaluate against a key. And those data are cheap. You can feed them into a machine that will then tell a student which video lecture she should watch next.

It’s cheap but it isn’t good. Students don’t enjoy it (see the section titled “Beyond the Videos“) and, pace Einstein, it turns out that just because it can be counted easily and cheaply doesn’t mean it counts. These startups take feeble data and pile a year’s worth of personalized recommendations on top of them. But feeble data return too many false positives (they claim a student knows something she really doesn’t) and too many false negatives (they claim a student doesn’t know something she really does). The personalization fails and so do the students.

So what’s good?

What’s good is giving students constructed-response tasks that require them to empty out the contents of their head, exposing all kinds of misconceptions about math to a trained educator’s eye who can recognize them and help.

But educators are expensive. And a lot of teachers don’t have the training or interest in gathering those kind of data, anyway, which results in the worst of both worlds: bad, expensive data.

Is there a third way here? Good, cheap data? ActiveGrade is interesting. It features a similar teacher dashboard to Khan Academy. The evaluations are made by a student’s teacher, not a machine, though, so it allows for better, more accurate personalization, if also more expensive.

It would be an interesting experiment to scan and send lots of class’ constructed responses to the same trained math educator who could evaluate those hundreds of assessments at a desk for an hour. I’m not volunteering and I’m not saying I prefer that hypothetical option to a trained educator’s understanding of her own students’ understanding. But I prefer that hypothetical option to the current reality of personalized math education today.