Space: Dan Meyer On Personalization

Here’s Einstein:

Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.

So what can computers count? The enthusiasm for personalized learning (one of the “trends to watch” in our assignment description) requires we answer that question.

If you’re Netflix, personalization means you know a) what movies a customer has rented and b) how she’s rated those movies on a scale from one to five. That’s what can be counted. The system counts it and then does its best to recommend another movie you’ll like. Netflix does a good job with those two data points, but it’s far from perfect.

Netflix might do a better job personalizing your movie-watching experience by having you type in a brief paragraph explaining your ranking. (Did you like a particular actor? Did one of the underlying themes resonate with you?) Then a Netflix employee would read that information and send you a recommendation.

I’m not saying that’s a good idea. That kind of personalization would be very expensive, for one. But that’s the axiom.

Good personalization is expensive.

Many math education startups (including those that have caught the eye of our instructors) have already met the challenge of cheap personalization, but good personalization still eludes them. Cheap personalization is easy if you look at math from the right angle. From that angle you see a lot of binary, right-or-wrong answers. You see multiple choice questions. You see text fields that are easy to parse as integers and then evaluate against a key. And those data are cheap. You can feed them into a machine that will then tell a student which video lecture she should watch next.

It’s cheap but it isn’t good. Students don’t enjoy it (see the section titled “Beyond the Videos“) and, pace Einstein, it turns out that just because it can be counted easily and cheaply doesn’t mean it counts. These startups take feeble data and pile a year’s worth of personalized recommendations on top of them. But feeble data return too many false positives (they claim a student knows something she really doesn’t) and too many false negatives (they claim a student doesn’t know something she really does). The personalization fails and so do the students.

So what’s good?

What’s good is giving students constructed-response tasks that require them to empty out the contents of their head, exposing all kinds of misconceptions about math to a trained educator’s eye who can recognize them and help.

But educators are expensive. And a lot of teachers don’t have the training or interest in gathering those kind of data, anyway, which results in the worst of both worlds: bad, expensive data.

Is there a third way here? Good, cheap data? ActiveGrade is interesting. It features a similar teacher dashboard to Khan Academy. The evaluations are made by a student’s teacher, not a machine, though, so it allows for better, more accurate personalization, if also more expensive.

It would be an interesting experiment to scan and send lots of class’ constructed responses to the same trained math educator who could evaluate those hundreds of assessments at a desk for an hour. I’m not volunteering and I’m not saying I prefer that hypothetical option to a trained educator’s understanding of her own students’ understanding. But I prefer that hypothetical option to the current reality of personalized math education today.

8 thoughts on “Space: Dan Meyer On Personalization

  1. I’m doing mastery based grading in my Alg 2 and Geometry classes. I was dismayed by the results on a quiz with simple (free response) questions on: Calculating slope and stating whether the line would rise, fall etc., graphing using slope and y-int, and graphing using x- and y-intercepts. I toyed with the idea of asking a colleague to grade one period of quizzes, just to see if they agree with my assessment.

    I find I now sit around with colleagues and hear them talk about their students getting this percent or that. I cringe, thinking “but do you know whether they know or don’t know: a) what the formula for slope IS, b) know the formula but don’t know what to do with the x’s and y’s, c) can’t correctly subtract and divide integers, d) what it means to rise or fall, why a slope of 2 is steeper than a slope of 1 etc. etc. At a glance, I can tell most of those things. And following it up with a few quick words with the student, I can point them in the right direction.

    But the time it takes…huge! (I’ve flipped Alg 2 classes.)

  2. Dan, your last paragraph is not crazy at all, in fact I feel it leads to an evaluation of not only the student, but also the teacher’s technique. Perhaps you should examine the procedures used to grade Advanced Placement Exams. This was my first year serving as a reader for the Statistics exams, and I found the process to be a profound professional development experience. I found myself not only looking at how students crafted their individual responses, but also noticed patterns in how some schools were training and/or developing their students. The quality of written expression, and the differences between students who could present a mathematical argument, as opposed to students who had mastered “canned” responses was clear. For me, the experience caused me to reflect upon my classroom practices, and whether those practices encouraged individual thought, or just led student to the solutions I desired. I left wishing I could provide feedback to teachers, stressing the positive and negative aspects of the responses I observed; and I also left wishing there were mechanisms for this sort of reflective grading which you describe for courses beyond AP.

  3. One reason I miss having my own classroom of students (in grad school right now) is exactly what you describe in the last paragraph. Maybe that means I’m crazy. I’m with Malisa, in that I don’t understand leaving student work unmined for formative assessment data.
    Have you heard of Autotutor? http://www.autotutor.org/ I’m not sure what’s happening with it right now, but the idea is to dialogue with students in a way that addresses their current thinking and the desired understanding. It’s sort of the same idea of how essays can be machine scored.
    Another angle: Are you assuming that the questions or prompts students are asked to respond to are of the type that would elicit information about the student’s understanding? (I don’t think you are, and I’m not sure if your assignment calls for a discussion of that.)

  4. Dan,

    I have been thinking a lot about this issue lately probably because I find that I don’t have the time to do my students justice when it comes to accurately diagnosing their understanding and then addressing their individual needs. I just gave my students an assessment today over place value in the fifth grade. I wish so much that I could then input the data I have into some program that would accurately assign work based on individual needs. Like a previous poster, I am currently trying to convert to mastery grading. It’s not easy to do in part because other teachers, parents and students have not been exposed to the idea. I certainly have no desire to score the items either. But after serving as a math interventionist, trained by Kentucky Center for Mathematics, I feel more qualified to assess student mathematical understanding than most teachers I work with because they simply have received no training for the task. We are currently being bombarded with computer programs that are supposed to make our jobs easier because they assess student understanding and then create new assignments. I have no confidence in any of these programs.

  5. I worked on a research project about personalization some time ago tracking issues around technology and personalization, and yes, things are more complicated than the cheap mass personalization would have us believe: http://lapl.wordpress.com.

    But I think you fall into the opposite trap of assuming that there is some perfect learning and knowledge that can be acquired with the help by some perfect experts (end even when this perfection is never achieved, it is there to be aspired to). Millions of people grew up and will continue to grow up without any deeper personal engagement with the subject or any competent feedback. Yet, we continue producing experts in all those subjects. When you ask these experts, you will find that their routes to expertise were very individual and circuitous – the only thing they share in common is the amount of time and effort they spent getting there. Yet, when you talk about personalization, you seem to want everybody to end up in the same place by pretty much the same route.

    Too often, personalization is simply equated with choosing when to do what kinds of exercises or maybe picking a particular approach and getting some individual feedback. But that doesn’t sound very personal to me. Personalization (or self-directedness as it used to be called not too long ago) should start with choosing to learn and choosing to learn something. As long as you have the tyranny of subjects and standards, you can’t really personalize much of anything. When you say personalization in this context, it’s like putting a gun to somebody’s head and saying would you rather eat worms or insects? Sure, personal choice is involved but not much freedom.

    But when people really choose what to learn, the results don’t look much like what learning looks like now. They most don’t choose to read Shakespeare or do physics. But they may choose things that may ultimately lead them to do both. The starting point of personalization must be that not everybody needs to do all subjects or do the subjects they do in the same order. Or just do them once.

    That’s what’s expensive about good personalization. Not the presence of some illusory expert educator to give feedback but the presence of true personal options with true personal consequences.

    This also does not mean a lack of rigour and discipline. But that only comes as a consequence of personalization not as some sort of a precondition. If I choose to become a plumber or a doctor, I need to learn a lot of things about physics and chemistry. And I need to learn them well. And it’s going to be hard and time consuming. And there will be people coming along saying, this would all have been so much easier if you’d only learned algebra in 6th grade. Maybe, but as long as people have opportunities to learn what they need when they need it, we can offer them true personalized education – education that is more like mentoring and apprenticeships. Because the truth is, all worthwhile knowledge and skills are developed that way anyway. Everything else is just really expensive pretending.

  6. Even the automatic system is expensive: it cost 1 million $ to improve Netflix’s algorithm for a 10.06% improvement in 2009:
    http://www.netflixprize.com/leaderboard

    “On September 21, 2009 we awarded the $1M Grand Prize to team “BellKor’s Pragmatic Chaos”. Read about their algorithm, checkout team scores on the Leaderboard, and join the discussions on the Forum. We applaud all the contributors to this quest, which improves our ability to connect people to the movies they love.”

    And I think they did that in 2007 and 2008 too.

  7. Dan, I don’t think your last idea is a crazy one, but like the scoring of state writing exams, I think it ultimately wouldn’t bring you good results. The people who score state writing assessments spend something like 60 seconds per essay. They look for key items and score without actually even reading the essay usually.

    When a classroom teacher takes time to look at an assessment, we bring a lot more to the table in terms of our knowledge of the student, his/her past performance, even his/her handwriting style.

    One solution is a less is more approach, using assessments that require deep evaluation at key points and relying on peer or self assessment at other times. I’ve had success with sitting a student down with their work, a rubric, and a key– I think they are harder on themselves than I am!

  8. Pingback: This Week’s “Links I Should Have Posted About, But Didn’t” | Larry Ferlazzo’s Websites of the Day…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>