Welcoming our robot grading overlords

Well, no, not really– but I thought that post title might be provocative, sort of like writing essays in a way that tricks the grading software.

There’s been a lot of discussion on things like WPA-L and elsewhere about “robo-readers,” as this New York Times piece sums up well, “Facing a Robo-Grader? Just Keep Obfuscating Mellifluously” and this further discussion on Slashdot. The very short version is it turns out that machines are just as capable of scoring writing that is completed as part of standardized tests– things like the GRE or SAT or other writing tests that ask students to respond to a very specific prompt.  Writing teachers of various flavors– the WPA-L crowd in general and Les Perelman from MIT in particular– are beside themselves with the wrongness of this software because it’s not human, it can be fooled, and cannot recognize “truth.”

Of course, ETS and Pearson (two of the companies that have developed this software) point out that they don’t intend this software to replace actual human feedback, that they admit this is not a way to check facts, and the software is not a good judge of “poetic” language.  And I’ve also seen plenty of humans fooled by untruths in print.  But never mind that; writing teachers are angry at the machine.

Now, I mostly (though obviously not entirely) agree with my WPA-L colleagues and Perelman, and, as I wrote about in my previous post, I’m not a fan of education that eliminates teaching and minimizes the opportunity for learning simply to jump through a credentialing loop.  So yes, I would agree that taking a batch of first year composition papers and dumping them into the robo-reading hopper to assign grades would a) not work and b) be bad.  Though again, it also appears that the people who have developed this software have the same position.

But let’s just say– hypothetically, mind you, and for the sake of argument– that this kind of software and its inevitable improvements might actually not be evil.  How might robo-grading (or maybe more accurately automated rating) software actually be “good?”

For starters, if this software is used in the way that ETS and Pearson say they are intending it to be used– that is, as a teaching/learning aid and not a grading tool per se– then it seems to me that this might be potentially useful along the lines of spellchecks, grammar-checks, and readability tests.  Is this a replacement for reader/ student/ teacher/ other human interaction in writing classes of various sorts?  Obviously not.  But that doesn’t mean it can’t be useful to readers, particularly teachers during the grading process.

Let’s face it:  the most unpleasant part of teaching writing is grading– specifically “marking up” papers students turn in to point out errors (and in effect justify the grade to the student) and to suggest ideas for revision.  It is very labor-intensive and the most boring part of the job, as I wrote about in some detail last year here.  If there was a computer tool out there that really would help me get this work done more efficiently and that would help my students improve, then why wouldn’t I use it?

Second, I think Perelman’s critique about how easily the machine is fooled is a little problematic– or at least it can be turned on its head.  It seems to me that if a student completing some standardized test writing is smart enough to out smart the machine– as Perelman demonstrates here— then perhaps that student actually does deserve a high grade from the machine.  It’s kind of like Kirk reprogramming the “no win” Kobayashi Maru test so he could win, right?

Third– and this is maybe something writing teachers in particular and writers in general don’t want to accept– writing texts that are well-received by machines is a pretty important skill to master.  I know that’s not the intention of this robo-reading software, but my writing teacher colleagues seem to suggest that this is not only an unnecessary skill but a particularly dangerous one.  Yet there is an entire web business called Search Engine Optimization that is (in part) about how to write web pages to include frequently searched keyword phrases so that the results appear higher in search engine– e.g., machine– results.  The keywords and structure of your monster.com resume can be half the battle in getting found by a potential employer who is using searches– e.g., machines– to find a match.

Anyway, you get the idea.  No, I don’t think we ought to turn over the teaching/grading function in writing classes to machines, and I don’t think a robo-grader is going to be able to look into the soul of the writer to seek Truth anytime soon.  But I think the blanket dismissal of and/or resistance to these kinds of tools by writing teachers is at best naive.  It’s probably more useful to imagine ways these tools can help our teaching practices in the long run.

6 thoughts on “Welcoming our robot grading overlords”

  1. Just wondering if one of those skills isn’t in resume writing when keywords facilitate getting past robo-readers. How else to get a job when they’re the gatekeepers? Do we need to teach our students this, too?

  2. The real issue is the robot writing which is the norm in FYW and college coursework in general. Machine-grading for machine-writing. What’s sad is many of the folks who encourage this garbage aren’t machine-teaching. They think what they’re doing is effective.

    1. This is very true Bradley, but it’s a tough cycle to break. It’s hard for teachers to teach things in fycomp that isn’t machine in, and it’s hard for students in fycomp to break out of the machine-writing habits. “Why don’t you research something you care about?” I ask. “I don’t care about anything,” I sometimes unfortunately hear in reply.

      Since I’ll be teaching fycomp in the first part of the summer term, these things are on my mind lately. And I am tempted to give assignments where students try to write like machines and/or to machines as some of the exercises. It’d be interesting to give them that Perelman piece he wrote to fool the robo-reader and to find out if they can do as well or better.

  3. What’s missing in this debate on both sides is one crucial bit of clarification. It’s not “grading” that’s hard, it’s producing a meaningful response to student writing that can

    a) help writers understand what revisions would improve their writing
    b) help students learn and improve in future writing situations
    c) help programs understand where the curriculum is working, where teachers can improve, etc.

    Even in a summative evaluation, a quality response can do all of these things. The grade is merely a comparative bit of data that is relatively easy to assign consistently with a little calibration (whether you are a human or a machine).

    1. Right: and what you’re describing here Bill (at least with a & b) is what I would call “teaching,” too. Of course, it also takes a little learning– that is, students have to want to revise, they have to want to make the connection to other courses and situations, etc.

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.