Category: AI Writing

Posted on

Four Rules For Discouraging Cheating with AI in Writing Classes

An indirect but positive review of Mollick’s “Co-Intelligence”

This semester, I’m teaching two sections of first year writing (aka freshman comp) and an advanced writing course called Digital Writing, and both have AI elements and themes. In first year writing, the research theme is “Your Career and AI.” In the Digital Writing course, the last two writing projects are going to be waist-deep in writing with AI. Maybe one day I will better understand/make use of Substack’s newsletter function to chronicle these classes in more detail, but that’s later.

For Digital Writing, we’re reading and discussing Ethan Mollick’s Co-Intelligence: Living and Working with AI. If you’re reading posts like this because you too are trying to make sense out of the what AI is about, there’s a good chance you’ve already heard of Mollick’s book and his Substack, One Useful Thing. If you haven’t heard of Mollick and you want to know more about AI but you’re overwhelmed with the firehose of news and information, then his book is for you. Co-Intelligence is a well-written, accessible, and a thoroughly researched 30,000 foot overview in less than 250 pages printed in a big font. It’s enough to get the “AI curious” up to speed on the current state of things (it was published in April 2024), while also pointing readers to ideas for further reading and research.

Mollick is a business professor at the Wharton School at the University of Pennsylvania, so he is primarily interested in how AI will impact productivity and innovation. I think we conceptualize teaching a bit differently, and like everything I’ve read about AI, Mollick is making some claims I doubt. But we’re mostly on the same page.

One of the most cited/mentioned chapters in Mollick’s book is “Four Rules for Co-Intelligence.” In brief, those rules are:

  • Always invite AI to the table, meaning you have to experiment and try to use AI (or really, different platforms, so AIs) for lots and lots of different things in order to discover what it/they can and can’t do.

  • Be the human in the loop: it’s a bad idea to completely turn over a task to AI, both because AI makes a lot of mistakes (aka hallucinations) and humans ought to be in charge rather than the other way around.

  • Treat AI like a person (but tell it what kind of person it is). AI doesn’t behave same way as other computer applications, so Mollick says we need to be conversational with it as if it were a human. Mostly he’s talking about creating context and scenarios in AI prompts, as in “You are an experienced teacher speaking to skeptical students about the value of group work. What advice would you give those students?”

  • Assume this is the worst AI you will ever use, which is perhaps the most accurate of these AI rules.

So, in that spirit, here are four more rules about teaching writing and AI— specifically, what teachers can do to discourage students from using AI to cheat.

Of course, I’m far from the first person to come up with four more rules for AI— I’m not even the first person to come up with four more rules for AI and writing! For example, there’s this fine post from Jane Rosenzweig at her site Writing Hacks “Four Rules for Writing in the Age of AI,” and also this guest post at John Warner’s Substack site by high school teacher and writer Brett Vogelsinger, “Artificial Intelligence and Writing: Four Things I Learned Listening to my High School Students.” Both great posts and great thoughts.

The most common concern about AI I read on Facebook (though not so much on Substack) from other professors and teachers is students using it to cheat on writing assignments. So this post isn’t about how to use AI to teach writing— maybe I’ll write more about that when I have a better sense of the answer. This is about how teachers can create an environment that discourages students from cheating with AI. It’s not foolproof. Sometimes, usually when they are desperate to try anything to pass the class, students cheat.

Teach writing as a process; don’t assign writing as a product.

I kicked off my writing about AI in this blog post from December 2022 “AI Can Save Writing by Killing ‘The College Essay.’” It’s the most frequently read post on the old blog. I wrote it in response to two different articles published in The Atlantic at the time arguing that the new ChatGPT had made writing assignments impossible and irrelevant.

Teaching writing as a process has been the mantra in composition and rhetoric since the late 1970s. Scholars debate the details about what this means, but in a nutshell, teaching writing as a process means setting up a series of assignments that begin with pre-writing invention exercises (freewriting and other brainstorming techniques, for example), activities that lead to rough drafts which are shared with other students through peer review. When students hand this work in, the instructor’s feedback is geared toward revision and (hopefully) improvement on future projects. My first year writing course is typical in that it is about research and students complete a research essay project. But long before we get to that assignment, students complete a series of smaller scaffolded assignments that build up to the larger essay. Again, none of this is new and it is how I was taught to teach writing back in the late 1980s when I started as a graduate teaching assistant.

I teach writing this way because there is good evidence that it works better than merely assigning writing. I also think teaching writing as a process deters plagiarism and other forms of cheating (including with AI). I require students to build their research writing projects through a series of smaller and specialized assignments, and to share their work in progress with other students in peer review. It’s awfully hard to fake this. Also, as I wrote back in July, I now make the process more visible by requiring students to complete their essays from beginning through final revisions on a Google Doc they share with me so I can view the document history and see what it is they did to put their writing together.

In contrast, assigned writing projects have always been much easier to cheat on. Before AI, students cheated with the internet, paper mills, by getting others doing the writing, or (at least according to my father who went to college in the early 1960s) with the library of papers that fraternities kept on hand.

There’s also the issue of the purpose of writing assignments in the first place. Teaching writing as a process is especially important in a course where the subject itself is writing and there is a lot of attention to how students craft their sentences and paragraphs. I realize that’s different from a class where the subject is literature or political science or business administration. But besides the fact that we should teach (not just assign) writing across the curriculum, writing assignments should ask students what they think about something. In research-based courses like freshman comp, students write about the research they did to persuade and inform both me and their classmates about something. It’s one of the reasons why I like teaching this class: my students are always teaching me new things. In my classes that are not as research-based (like Digital Writing), students write and reflect on the assigned readings and other projects of the class in order to share with readers what they think.

Assigned writing tasks tend to seek specific answers based on the content of the course— write about the theme of madness in Hamlet, about the balance of power between the three parts of the federal government, about they key causes of the great recession, etc. In evaluating assigned writing, teachers are less interested in what students think and are more in seeing if students correctly repeated the content of the course the teacher delivered through lectures, activities, and readings. In other words, assigned writing is an assessment tool, like an exam— and in most cases, it probably would be a more effective to use an exam.

Now, teaching writing as a process is A LOT more work for everyone because it means more reading, more teacher commenting, and more checking in with students’ writing as they progress through these assignments. This is why at the vast majority of colleges in the U.S., first year writing courses have 25 or fewer students. Some colleagues who teach lecture courses with 100 or so students who also assign papers have asked me how they’re supposed to teach writing as a process in these courses. My answer is I wouldn’t. Instead, I’d rely on short written responses to readings, quizzes, and exams.

Any course assignment that could be completed without being present in that course is a bad assignment.

A lot of the hype around AI is about how great it is at passing tests— LSAT, GRE, SAT, etc. etc.— and how that is supposed to mean something. But besides the issue of whether AI can pass these tests because it “knows” or because the test questions were part of the content used to create the AI, I think we all know this is not how school works. I mean, if on the first day of a course I introduced all the writing assignments, and then a student showed up on the second day and said “I finished everything— can I get my A now?” the answer, obviously, is no.

Which brings me to this second rule: if a teacher gives students an exam or an assignment that could be successfully completed without ever being in the class, then that’s a bad assignment. This is something I never thought about before AI. In the old old days, I don’t think it made much difference. When I went to college in the mid 1980s, if someone could pass an intro to chemistry exam or a history 101 exam without ever attending the class, what’s the problem? They already had enough mastery of the subject to pass the class anyway. That started to end with students doing Google searches to pass exams, and now that AI can answer all those questions in that history 101 class final in real time, it’s completely over.

AI isn’t attending classes with our students (at least not yet), and so it is not as useful to cheat on exams or assignments that have specific connections to the course. That’s easy enough to do in the kinds of courses I teach, though I have to assume this is more complicated in a subject like calculus where the concepts and methods transcend classroom boundaries. But perhaps an even easier way to address this problem is for the teacher to make participation count as part of the grade. As I discussed in this post, my classes have a participation grade component that counts for about 30% of the grade.

AI detection software doesn’t work and it never will.

A lot of teachers want to skip these first two rules and instead just rely on some kind of app that can detect what parts of a student’s paper were written by an AI. Essentially, they want something like the plagiarism detection software Turnitin many of these teachers have used for years. Though as a quick glance at the Turnitin website reveals, they are shifting from plagiarism detection along with AI detection as well.

Plagiarism detection software has been a divisive topic in writing studies for years. While I know lots of teachers routinely require their students to run their papers through Turnitin for a plagiarism check, I never have done this because I don’t think it’s necessary and I don’t think Turnitin is as good of a tool as many users seem to think. This is especially true with AI detection. According to Turnitin, the false-positive rate for “fully human-written text” is less than 1%, but up to 20% for AI writing. And that is just for the very common and very dumb way people use to cheat with AI: writing a simple prompt and copying and pasting the answer with few changes. I have to assume the ineffectiveness in detecting AI goes down if the human using the AI effectively: for brainstorming, proofreading/editing, chatting with it about revision ideas, and so forth.

It’s a futile effort, especially as the AIs improve and as all of us (including our students) learn more about how to use them for not just cheating. Which leads me to my last point:

Teachers at all levels need to learn more about AI.

Colleges and universities are certainly trying. The two talk things I did last year about AI were both faculty development events, and the attendance at both was pretty good. I know folks here at EMU have held similar events, and I get the impression this is pretty common at most colleges and universities. And faculty have heard of AI at this point, of course.

The problem is I’m not sure any of the faculty development or the oodles of news stories about AI has resulted in any differences in teaching. This is mostly just based on my own sense of things, but I did informally poll my current students (I have about 70 this semester) the other day about AI in other classes they were taking. A few students mentioned classes where they are using AI for various assignments. A few other students mentioned instructors who expressly forbid the use of AI. I asked these students if they thought the instructor had any way of enforcing that; “no.” But the majority of my students said that the topic has not come up at all. That’s a problem.

I’m not saying every teacher now needs to embrace AI and incorporate it into their teaching. Not at all. Besides experimenting with AI in my teaching, I’ve been doing a lot of writing and reading about AI that is (hopefully) going to turn into a research project. I think my teaching with AI experiments are going well, but I honestly don’t know if this is something I’ll continue to do in the future. I feel the same way about AI generally: it probably is going to “change everything,” but it also might end up being another one of those things (like MOOCs, which was the subject of my last major research project) that never lives up to the hype.

What I am saying though is AI is here now and it looks like it’s going to be (probably) a big deal for some time to come. It is not just going to “go away” and it cannot be ignored. A professor or teacher can continue to refuse to engage with AI for valid ethical or personal reasons, but that is not going to stop everyone else from using it. That includes some of our students who are using AI simplistically to cheat, perhaps by feeding the teacher’s writing assignment into ChatGPT and copying/pasting whatever the AI comes up with. Fortunately, it’s pretty easy to spot that sort of AI use. But what teachers cannot easily recognize or stop is a student who uses AI more in the way that it is really meant to be used: as a tool to help/improve what humans do, not replace it.

So start learning about AI, even if you hate it. Mollick’s book is a good place to start.

Posted on

A Small Example of the Jagged Frontier and Discovering What ChatGPT Can’t Do

Or, the case of the penless writing logo

One of the things about AI I find most fascinating (and I read about this in other commentaries about ChatGPT and other AIs all the time) is it is really good at lots of things and weirdly bad at lots of other things, and that seems especially true with a lot of things humans do very well. In his book Co-Intelligence: Living and Working with AI, Ethan Mollick describes this as the “jagged technological frontier.” He describes a large study he and some colleagues did (I will probably need to read this and possibly it assign this fall), and in the abstract of that study, they write that AI creates a jagged technological frontier “where some tasks are easily done by AI, while others, though seemingly similar in difficulty level, are outside the current capability of AI.” ChatGPT can easily make a pretty good looking (to me) logo, but it cannot separate writing from the word/image/concept of “pen.”

This is going to find its way into my courses this semester; as one of a series of exercises, I want to ask students to discover for themselves something the AI can’t do, and, if possible, make a guess as to why. I guess this would be my example of some of that assignment. I had a “conversation” with ChatGPT the other day to create a logo for the Canvas site for my Digital Writing class this fall. This is using the “not free” version which allows me to generate lots of images (I think the free version limits users to just two a day.). ChatGPT won’t let me just share the conversation, so I’ll reproduce/paraphrase it here.

I started with this prompt:

“Draw me a logo for the top of a website for a college class I’m teaching called digital writing. The class examines writing as a technology. Since the course number is 354, incorporate the number somewhere into the design. The university’s colors are green and white, but it is okay to incorporate other colors if you want.”

ChatGPT made this:

Then I asked it to make a JSON script that describes what this image looks like. I’m going to skip that in this post in part because I don’t know enough about JSON, and in part that’s not the point I’m trying to make. That’s maybe a future post.

Anyway, here’s an image it made with the JSON script:

And here’s another one it made when I asked it to repeat what it did before:

FWIW, this was my favorite and the one I went with— after some minor editing.

Then I wrote: “Now forget the json script. Instead, draw me a logo for the top of a website for a college class I’m teaching called digital writing. The class examines writing as a technology. Since the course number is 354, incorporate the number somewhere into the design. The university’s colors are green and white, but it is okay to incorporate other colors if you want.” Here’s what it drew:

To me, I think the the reproduction of the logo by ChatGPT is getting a little worse in some ways. I mean, “Nourse?”

Then I wrote: “I noticed you added a pen even though that wasn’t in the prompt. There are a lot of other kinds of writing implements and tools. Why did you decide on using a fountain pen?”

ChatGPT responded: “I chose to include a fountain pen because it’s a classic symbol of writing, representing the traditional side of the writing process. The contrast between the fountain pen and digital elements (like circuits or binary code) visually communicates the course’s theme of examining writing as a technology.

“Fountain pens are often seen as timeless and elegant, bridging the gap between old and new, which fits the idea of exploring writing in both traditional and technological contexts. It also has a distinctive and recognizable shape, which makes it effective in a logo.

“If you’d prefer a different writing tool or symbol, I can certainly create a version with an alternative element!”

I wrote: “How do you think it might look if instead of a fountain pen you used a typewriter instead?”

ChatGPT responded: “Using a typewriter in the logo could create a strong visual connection to the history of writing technology, emphasizing the transition from traditional mechanical tools to digital ones. The typewriter could be depicted with keys or as a whole, evoking a sense of retro charm while still aligning with the theme of writing as technology. This could give the logo a more nostalgic feel, yet still relevant to the course’s examination of the evolution of writing tools.

“If you’d like, I can create a version of the logo with a typewriter instead of the fountain pen to see how it looks. Would you like me to do that?”

“Sure, create a version like that” I wrote back. Here’s what it made:

ChatGPT also wrote with this image “Here is the logo with a typewriter element instead of a fountain pen. This version emphasizes the traditional aspect of writing through the typewriter while maintaining a connection to digital technology. If you have any further requests or adjustments, feel free to let me know!” Oh, the mansplaining confidence of ChatGPT, trying to convince me there was no pen anymore!

I responded “That’s interesting, but there is still a pen in the logo. can you eliminate that and the three smaller versions of the logo on the left side and in the right corner?” It produced this:

And ChatGPT wrote “Here is the revised logo, with the pen removed and only the typewriter elements included. The smaller versions of the logo have also been eliminated. Let me know if this version meets your needs or if you would like any further adjustments!” Again, it seems to be glitching more on some of the words and such.

I responded “Nope, the pen is still there and there are still three smaller logos, this time across the bottom of the page. Can you get rid of the pen and the smaller logos?” ChatGPT made this:

I find the inclusion of a photo-realistic pen/mechanical pencil particularly amusing, like ChatGPT is just trying to sneak it in there. You can also see where it’s getting more confused with “W- – +—33ING” or whatever that is. I responded “The small logos are gone, but there is still a writing implement– a pen or a mechanical pencil, I can’t tell. Also you the writing for digital writing and university at the bottom of the logo shouldn’t be there. Can you eliminate these things?” It made this:

It also wrote back “This version removes the pen or pencil, as well as the additional text at the bottom of the logo. The design now focuses entirely on the typewriter and digital elements, with the course number 354 subtly incorporated. If this meets your expectations or if further adjustments are needed, just let me know!” It’s almost as if it is admitting that yeah, the pencil/pen is still there, but at least the typewriter is what is being emphasized.

Here’s the punchline for me. I asked again “Can you delete the pencil, the pen, and the word University from this logo?” ChatGPT’s response surprised me:

“It seems that the tools available to me do not allow direct editing of existing images to remove specific elements like the pencil, pen, or text. Instead, I can generate a new logo without those elements, as I attempted to do in the previous steps. However, if those elements persist in the new generation, it’s due to the constraints of the image generation process.

“If you have access to image editing software, you might be able to manually remove the undesired elements. Alternatively, I can keep trying to refine the prompt to produce a version that meets your specifications. Would you like me to attempt another version or provide instructions for editing the image manually?”

I asked “Can you tell me more in simple layman terms what you mean by the constraints of the image generation process?”

ChatGPT answered “Sure! When I generate images, I’m using a process that interprets a text description to create a visual design. The tool I use is powerful but not perfect—it tries to match the description as closely as possible, but sometimes it includes extra elements or doesn’t fully understand specific requests, like leaving out certain objects or text.

“These ‘constraints’ mean that even when I specify not to include a pen or extra text, the tool might still add them because it’s interpreting the description in a way that makes sense to it, based on patterns it has learned from millions of images.

“So, while I can describe what you want very precisely, the image might still include some unwanted elements because the generation process doesn’t always get it exactly right. This is why there might be some back-and-forth to refine the result, but it’s not always guaranteed to be perfect due to these limitations.”

If I am understanding this correctly, I think what I am running into here is a bit of that jagged frontier and the limits of the AI’s training data. Essentially, I think ChatGPT’s association between images of writing and pens/pencils are so connected that it can’t make a picture of writing that does not include a pen or a pencil.

When you think about it for a second, this isn’t too surprising. Try doing a Google image search for just the word “writing” to see what I mean. Here’s a screenshot of what I came up with from such a search:

It’s no wonder that ChatGPT thinks that the idea/word/concept “writing” goes with pen or pencil because of the rest of the data it’s been trained on does the same thing. Which is to say I think when you ask a human to create an image of the process of writing, they default to a pen.

Though I will say ChatGPT’s advice for me the human in this loop: why don’t you just use some image editing software and fix it yourself?

Posted on

Why Teaching Citation Practices (yes, I’m talking MLA/APA style) is Even More Important with AI

A couple weeks ago, I wrote about why I use Google docs to teach writing at all levels. I’ve been using it for years–long before AI was a thing–in part because being able to see the history of a student’s Google doc is a teachable moment on the importance of the writing and revision process. This also has the added bonus of making it obvious if a student is skipping that work (by using AI, by copying/pasting from the internet, by stealing a paper from someone else, etc.) because the document history goes from nothing to a complete document in one step. I’m not saying that automatically means the student cheated, but it does prompt me to have a chat with that student.

In a similar vein and while I’m thinking about putting together my classes for the fall term, I thought I’d write about why I think teaching citation practices is increasingly important in research writing courses, particularly first year composition.

TL;DR version: None of this is new or innovative; rather, this is standard “teaching writing as a process” pedagogy and I’ve been teaching research writing like this for decades. But I do think it is even more important to teach citation skills now to help my students distinguish between the different types of sources, almost all of which are digital rather than on paper. Plus this is an assignment where AI might help, but I don’t think it’d help much.

Continue reading “Why Teaching Citation Practices (yes, I’m talking MLA/APA style) is Even More Important with AI”

Posted on

Why I Use Google Docs to Teach Writing, Especially in the Age of AI

I follow a couple different Facebook groups about AI, each of which have become a firehose of posts lately, a mix of cool new things and brand new freakouts. A while back, someone in one of these groups posted about an app to track the writing process in a student’s document as a way of proving that the text was not AI. My response to this was “why not just use Google docs?”

I wish I could be more specific than this, but I can’t find the original post or my comment to it; maybe it was deleted. Anyway, this person asked “what did I mean?” and I explained it briefly, but then I said I was thinking about writing a blog post about it. Here is that post.

For those interested in the tl;dr version: I think the best way to discourage students from handing in work they didn’t create (be that from a papermill, something copied and pasted from websites, or AI) is to teach writing rather than merely assigning writing. That’s not “my” idea; that’s been the mantra in writing studies for at least 50 years. Also not a new idea and one you already know if you use and/or teach with Google docs: it is a great tool for teaching writing because it helps with peer review and collaborative writing, and the version history feature helps me see a student’s writing process, from the beginning of the draft through revisions. And if a student’s draft goes from nothing to complete in one revision, well, then that student and I have a chat.

Continue reading “Why I Use Google Docs to Teach Writing, Especially in the Age of AI”

Posted on

No, Student Writing Is Not Dead (or how AI faculty freakout is back)

Now that the 2023-24 school year is long over and my wife and I are (mostly) done moving into our new house, it’s time to start thinking again about AI for teaching in the fall and for some scholarly things beyond. I’ve been mostly ignoring these things for the last couple of months, but even in that short time, it feels like things have changed. AI tech is getting quickly integrated into everything you can imagine, and it feels to me like the AI faculty freakout factor is on the rise once again.

This is just a gut feeling– like I said, I’ve been out of the loop and it’s not like I’ve done any research on this. But the current moment reminds me a bit of late 2022/early 2023 when ChatGPT first appeared. By the time I did a talk about AI at Hope College in late April 2023 and also again a talk/workshop about AI (over Zoom) at Washtenaw Community College in October 2023, teachers had settled down a bit.  Yes, faculty were still worried about cheating and the other implications, but I think most of the folks who attended these events had already learned more about AI and had started to figure out how to both use it as a tool to help teaching. They also realized they needed to make some changes to assignments because of AI tools.

But now the freakout is back. Perhaps it’s because more faculty are starting to realize that “this whole AI thing” is something they’re going to have to deal with after all. And as far as I can tell, a lot of the freaked out faculty are in the humanities in general/in English in particular. I suppose this is because we teach a lot of general education classes and classes that involve a lot writing and reading. But I also think that the reason why the freakout is high in fields like English is because a lot of my department and discipline colleagues describe themselves as being “not really into technology.”

The primary freakout then and now– at least among faculty in the humanities (I assume STEM faculty have different freakout issues)– is that AI makes it impossible to teach writing in college and in high school because it is too easy for students to have ChatGPT (or whatever other AI) to do the work for them. I wrote a post in response to these articles back in December 2022, but there were dozens of freakout articles like these two. These articles almost always assume that AI has uniquely enabled students to cheat on assignments (as if paper mills and copy and pasting from “the internet” hadn’t existed for decades), and that given the chance, students will always cheat. So the only possible solution is to fight AI with things like detection software or returning to handwritten exams.

It’s deja vu all over again.

Consider, for example, Lisa Lieberman’s June 2024 Chronicle of Higher Education article “AI and the Death of Student Writing.” Lieberman, who teaches community college English and composition courses “in California’s Central Valley,” has seen an alarming uptick in students using AI to write their papers. She gives an example of a student’s essay about The Shining that included the sentence “A complex depiction of Jack’s development from a struggling family guy to a vessel of lunacy and malevolence is made possible by Stanley Kubrick’s brilliant direction.” Lieberman writes “I called the student in and asked him to write a sentence with the word ‘depiction.’ He admitted he didn’t know what ‘depiction’ meant, much less how to spell it, much less how to use it in a sentence. He confessed he hadn’t written a single word of the essay.” (For what it’s worth, I would have asked this student about “malevolence”).

Then she moves on to discussing a student writing her essay with the now AI-fueled version of Grammarly. Lieberman “discovered it’s a multilayered computer program that does everything from simple spelling and grammatical corrections to rewriting entire sentences, adjusting tone and fluency.” She estimated that at least of a third of her students were consistently using AI: “Once they believed they could turn in AI assignments undetected, they got bolder … and used AI for every single assignment.”

It’s all just so wrong, Lieberman laments, in part because of how her students are just cheating themselves by using AI. Here’s a long quote from the end of the article:

I remember my days at Berkeley, where, as an English major, I’d take my copy of Wallace Stevens’s The Palm at the End of the Mind, or Chaucer’s “The Wife of Bath’s Tale,” and pick a nice, sunny spot on campus on a grassy knoll underneath a tree, lay out my blanket, and spend the afternoon reading and scribbling notes in my books. It was just me and my books and my thoughts. There was nothing better.

As I lay there reading the writer’s words, they came to life — as if the author were whispering in my ear. And when I scribbled my notes, and wrote my essays, I was talking back to the author. It was a special and deep relationship — between reader and writer. It felt like magic.

This is the kind of magic so many college students will never feel. They’ll never feel the sun on their faces as they lie in the grass, reading words from writers hundreds of years ago. They won’t know the excitement and joy of truly interacting with texts one-on-one and coming up with new ideas all by themselves, without the aid of a computer. They will have no idea what they’re missing.

I understand the anxiety that Lieberman is expressing, and I completely agree that AI technology is forcing us to change how we teach college classes– and, in particular, classes where students are expected to read and to write about that reading.

However:

  • Students have been cheating in school for as long as there has been school. AI make it easier (and more fun!) to cheat, but none of this is new. So any educator who thinks that students have only now started to cheat on the things they assign only because of AI are kidding themselves.
  • In my experience, the vast majority of students do not want to cheat this much. Oh sure, they might cheat by poorly borrowing a quote from a website, or looking over someone’s shoulder to get a quiz answer on a multiple choice test. But in my view, these are misdemeanor offenses at best. Also, when students do not cite sources properly (and this is as true for the MA students I work with as it is with the first year writing students), it’s because they don’t know how. In other words, a lot of plagiarism is a teachable moment.
  • Also in my experience, students who do blatantly cheat by downloading from a papermill or prompting an AI to do the whole assignment are a) already failing and desperate, and b) not exactly “criminal masterminds.” Every freakout narrative I’ve read– including Lieberman’s– includes a “scene” where the instructor confronts the student with the obvious AI cheating. So to me, if it’s this easy to catch students who cheat using AI, what’s the problem? Just punish these students and be done with it.
  • The fundamentals of teaching writing as a process– the mantra of writing studies for the last 50+ years– are still the same and the best way to discourage students from cheating with AI or anything else. Don’t merely assign writing– teach it. Make students show their work through drafts. Use a series of shorter assignments that build to a larger and more complex writing project. In a research-oriented writing class (like first year composition, for example), require students to create an annotated bibliography of all of their sources. Have peer review as a required part of the process. Etc., etc., etc. None of this is foolproof and for all I know, Lieberman is already doing this. But besides actually helping students to become better writers, teaching (rather than just assigning) writing like this makes cheating as much work as just doing the assignments.
  • I think the best way to dissuade students from using AI to cheat is to explain to them why this is a bad idea. Last year, I had a discussion at the beginning of all of my classes on the basics of AI and why it might be useful for some things (see my next bullet), and why it is not useful for cheating, and that’s especially true in classes that involve research and where writing is taught as a process (see my previous bullet). I think by making it clear from the beginning that yes, I too knew about AI and here’s why cheating with it isn’t a good idea, fewer of them were tempted to try that in my classes.
  • I don’t think there’s anything wrong with Grammarly. At EMU, I will often get letters of accommodation from the disability office about students enrolled in my classes that tell me how I am supposed to “accommodate” the student. That usually means more time to take exams or more flexibility for deadlines, but often, these letters say I should allow the student to use Grammarly.

My philosophy on this has always been that it is a good idea for students to seek help with their writing assignments from outside of the class–help that assists, not that does the work for the student. I always encourage students– especially the ones who are struggling– to get help from a writing center consultant/tutor, a trusted friend or parent, and so forth. I think Grammarly– when used properly– falls into that category. I don’t think asking Grammarly to write the whole thing counts as “proper use.” I want students to proofread what they wrote to make sure that the  mechanics of their writing are as clear and “correct” as possible, and if Grammarly or an AI or another electronic tool can help with that, I’m all for it.

I think the objection that Lieberman has with Grammarly is it makes writing mechanically correct prose too easy, and the only way for students to learn this stuff is to make them do it “by hand.” As someone who relies heavily on a calculator for anything beyond basic arithmetic and also as someone who relies on Google Doc’s spell checking and grammar checking features, I do not understand this mindset. Since she’s teaching in a community college setting, I suppose Liberman might be working more with “basic writing” students. I could see more of an argument for getting students to master the basics before relying on Grammarly. But for me and even in classes like first year writing, I want to focus mostly on the arguments my students are making and how they are using evidence to support their points. So if a student gets some help with the mechanics from some combination of a writing center consultant and an application like Grammarly, then I can focus more exclusively on the interesting parts.

Where Lieberman and I might agree though is if a student doesn’t have basic competency with writing mechanics, then Grammarly is not going to solve the problem. It’s a lot like the mistakes students still make with there/their/they’re even if they take the time to spell check everything. And again, that’s why it is is so easy to detect AI cheating: the vast majority of students I have had who have tried to cheat with AI have done it poorly.

  • Finally, about students missing “the magic” of reading and writing, especially while doing something clichéd idealistic like laying on a blanket on the campus lawn and under an impressive oak. I get it, and that’s part of why I went into this line of work myself. But this is the classic mistake so many teachers make: just because the teacher believes reading and writing are magical doesn’t mean your students will. In fact, in required gen ed classes like first year writing or intro to literature, many (sometimes most) of the students in those classes really do not want to take those courses at all. I can assign students to read a book or essay that I think is great or I can encourage students to keep writing on their own and for not just school, and sometimes, I do have students who do discover “the magic,” so to speak. But honestly, if the majority of my first year writing students at the end of the semester come away thinking that the experience did not “totally suck,” I’m happy.

So no, this is not the end fo student writing. It’s just different. Again.

Posted on

TALIA? This is Not the AI Grading App I Was Searching For

(My friend Bill Hart-Davidson unexpectedly died last week. At some point, I’ll write more about Bill here, probably. In the meantime, I thought I’d finish this post I started a while ago about the webinar about Instructify’s AI grading app. Bill and I had been texting/talking more about AI lately, and I wish I would have had a chance to text/talk more about this. Or anything else).

In March 2023, I wrote a blog post titled “What Would an AI Grading App Look Like?” I was inspired by what I still think is one of the best episodes of South Park I have seen in years, “Deep Learning.”  Follow this link for a detailed summary or look at my post from last year, but in the nutshell, the kids start using ChatGPT to write a paper assignment and Mr. Garrison figures out how to use ChatGPT to grade those papers. Hijinks ensue.

Well, about a month ago and at a time when I was up to my eyeballs in grading, I saw a webinar presentation from Instructify about their AI product called TALIA. The title of the webinar was “How To Save Dozens of Hours Grading Essays Using AI.” I missed the live event, but I watched the recording– and you can too, if you want— or at least you could when I started writing this. Much more about it after the break, but the tl;dr version is this AI grading tool is not the one I am looking for (not surprisingly), and I think it would be a good idea for these tech startups to include people with actual experience with teaching writing on their development teams.

Continue reading “TALIA? This is Not the AI Grading App I Was Searching For”

Posted on

Once Again, the Problem is Not AI (a Response to Justus’ and Janos’ “Assessment of Student Learning is Broken”)

I most certainly do not have the time to be writing this  because it’s the height of the “assessment season” (e.g., grading) for several different assignments my students have been working on for a while now. That’s why posting this took me a while– I wrote it during breaks in a week-long grading marathon. In other words, I have better things to do right now. But I find myself needing to write a bit in response to Zach Justus and Nik Janos’ Inside Higher Ed piece “Assessment of Student Learning is Broken,” and I figured I might as well make it into a blog entry. I don’t want to be a jerk about any of this and I’m just Justus and Janos are swell guys and everything, but this op-ed bothered me a lot.

Justus and Janos are both professors at Chico State in California; Justus is a professor in Communications and is the director of the faculty development program there, and Janos is in sociology. They begin their op-ed about AI “breaking” assessment quite briskly:

Generative artificial intelligence (AI) has broken higher education assessment. This has implications from the classroom to institutional accreditation. We are advocating for a one-year pause on assessment requirements from institutions and accreditation bodies. We should divert the time we would normally spend on assessment toward a reevaluation of how to measure student learning. This could also be the start of a conversation about what students need to learn in this new age.

I hadn’t thought a lot about how AI might figure into institutional accreditation, so I kept reading. And that’s where I first began to wonder about the argument they’re making, because very quickly, they seem to equate institutional assessment with assessment in individual classes (grading). Specifically, most of this piece is about the problems caused by AI (supposedly) of a very specific assignment in a very specific sociology class.

I have no direct experience with institutional assessment, but as part of the Writing Program Administration work I’ve dipped into a few times over the years, I have some experience with program assessment. In those kind of assessments, we’re looking at the forest rather than the individual trees. For example, maybe as part of a program assessment, the WPAs might want to consider the average grades of all sections of first year writing. That sort of measure could tell us stuff about the overall pass rate and grade distribution across sections, and so on.  But that data can’t tell you much about grades for specific students or the practices of a specific instructor. As far as I can tell, institutional assessments are similar “big picture” evaluations.

Justus and Janos see it differently, I guess:

“Take an introductory writing class as an example. One instructor may not have an AI policy, another may have a “ban” in place and be using AI detection software, a third may love the technology and be requiring students to use it. These varied policies make the aggregated data as evidence of student learning worthless.”

Yes, different teachers across many different sections of the same introductory writing class take different approaches to teaching writing, including with (or without) AI. That’s because individual instructors are, well, individuals– plus each group of students is different as well. Some of Justus and Janos’ reaction to these differences probably have to do with their disciplinary presumptions about “data”: if it’s not uniform and if it not something that can be quantified, then it is, as they say, “worthless.” Of course in writing studies, we have no problem with much more fuzzy and qualitative data. So from my point of view, as long as the instructors are more or less following the same outcomes/curriculum, I don’t see the problem.

But like I said, Justus and Janos aren’t talking about institutional assessment. Rather, they devote most of this piece to a very specific assignment. Janos teaches a sociology class that has an institutional writing competency requirement for the major. The class has students “writing frequently” with a variety of assignments for “nonacademic audiences,” like “letters-to-the-editor, … encyclopedia articles, and mock speeches to a city council” meeting. Justus and Janos say “Many of these assignments help students practice writing to show general proficiency in grammar, syntax and style.” That may or may not be true, but it’s not at all clear how this was assigned or what sort of feedback students received. .

Anyway, one of the key parts of this class is a series of assignments about:

“a foundational concept in sociology called the sociological imagination (SI), developed by C. Wright Mills. The concept helps people think sociologically by recognizing that what we think of as personal troubles, say being homeless, are really social problems, i.e., homelessness.”

It’s not clear to me what students read and study to learn about SI, but it’s a concept that’s been around for a long time– Mills wrote about it in a book in the 1950s. So not surprisingly, there is A LOT of information about this available online, and presumably that has been the case for years.

Students read about SI and as part of their study, they “are asked to provide, in their own words and without quotes, a definition of the SI.” To help do this, students do activities like “role play” to they are talking to friends or family about a social problem such as homelessness. “Lastly,” (to quote at length one last time):

…students must craft a script of 75 words or fewer that defines the SI and uses it to shed light on the social problem. The script has to be written in everyday language, be set in a gathering of friends or family, use and define the concept, and make one point about the topic.

Generative AI, like ChatGPT, has broken assessment of student learning in an assignment like this. ChatGPT can meet or exceed students’ outcomes in mere seconds. Before fall 2022 and the release of ChatGPT, students struggled to define the sociological imagination, so a key response was to copy and paste boilerplate feedback to a majority of the students with further discussion in class. This spring, in a section of 27 students, 26 nailed the definition perfectly. There is no way to know whether students used ChatGPT, but the outcomes were strikingly different between the pre- and post-AI era.

Hmm. Okay, I have questions.

  • You mean to tell me that the key deliverable/artifact that students produce in this class to demonstrate that they’ve met a university-mandated gen ed writing requirement is a 75 word or fewer passage? That’s it? Really. Really? I am certainly not saying that being able to produce a lot of text should not be the main factor for demonstrating “writing competency,” but this seems more than weird and hard to believe.
  • Is there any instructional apparatus for this assignment at all? In other words, do students have to produce drafts of this script? Are there any sort of in-class work with the role-play that’s documented in some way? Any reflection on the process? Anything?
  • I have no idea what the reading assignments and lectures were for this assignment, so I could very well be missing a key concept with SI. But I feel like I could have copied and pasted together a pretty good script just based on some Google searching around– if I was inclined to cheat in the first place. So given that, why are Justus and Janos confident that students hadn’t been cheating before Fall 2022?
  • The passage about the “before Fall 2022” approach to teaching this writing assignment says a lot. It sounds like there’s no actual discussion of what students wrote, and the main instructions to students back then was to follow “boilerplate feedback.” So, in assessing this assignment, was Janos evaluating the unique choices students made in crafting their SI scripts? Or rather, was he evaluating these SI scripts for the “right answer” he provided in the readings or lectures?
  • And as Justus and Janos note, there is no good way to know for certain if a student handed in something made in part or in whole by AI, so why are they assuming that all of those students who got the “right answer” with their SI scripts were cheating?

So, Justus and Janos conclude, because now instructors are evaluating “some combination of student/AI work,” it is simply impossible to make any assessment for institutional accreditation. Their solution is “we should have a one-year pause wherein no assessment is expected or will be received.” What kinds of assessments are they talking about? Why only a year pause? None of this is clear.

Clearly, the problem here is not institutional assessment or the role of AI; the problem is the writing assignment. The solutions are also obvious.

First, there’s the teaching writing versus assigning it.  I have blogged a lot about this in the last couple years (notably here), but teaching writing means a series of assignments where students need to “show their work.” That seems extremely doable with this particular assignment, too. Sure, it would require more actual instruction and evaluation than “boilerplate feedback,” but this seems like a small class (27 students), so that doesn’t seem that big of a deal.

Second, if you have an assignment in anything that can successfully be completed with a simple prompt into ChatGPT (as in “write a 75 word script explaining SI in everyday language”), then that’s definitely now a bad assignment. That’s the real “garbage in, garbage out” issue here.

And third, one of the things that AI has made me realize is if an instructor has an assignment in a class– and I mean any assignment in any class– which can be successfully completed without having any experience or connection to that instructor or the class, then that’s a bad assignment. Again, that seems like an extremely easy to address with the assignment that Justus and Janos describe. They’d have to make changes to the assignment and assessment, of course, but doesn’t that make more sense than trying to argue that we should completely revamp the institutional accreditation process?

Posted on

I’m Still Dreaming of an AI Grading Agent (and a bunch of AI things about teaching and writing)

I’m in the thick of the fall semester and I’ve been too busy to think/read/write much about AI for a while. Honestly, I’m too busy to be writing this right now, but I’ve also got a bucket full of AI tabs open on my browser, so I thought I’d do a bit of a procrastination and “round up” post.

In my own classes, students seem to be either leery of or unimpressed with AI. I’ve encouraged my more advanced students to experiment with/play around with AI to help with the assignments, but absent me requiring them to do something with AI, they don’t seem too interested. I’ve talked to my first year writing students about using AI to brainstorm and to revise (and to be careful about trusting what the AI presents as “facts”), but again, they don’t seem interested. I have had at least one (and perhaps more than that) student who tried to use AI to cheat, but it was easy to spot. As I have said before, I think most students want to do the work themselves and to actually learn something, and the students who are inclined to cheat with AI (or just a Google search) are far from criminal geniuses.

That said, there is this report, “GenAI in Higher Education: Fall 2023 Update Time for Class Study,” which was research done by a consulting firm called Tyton Partners and sponsored by Turnitin. I haven’t had a chance to read beyond the executive summary, but they claim half of students are “regular users” of generative AI, though their use is “relatively unsophisticated.” Well, unless a lot of my students are not telling me the truth about using AIs, this isn’t my impression. Of course, they might be using AI stuff more for other classes.

Here’s a very local story about how AI is being used in at least one K-12 school district: “‘AI is here.’ Ypsilanti schools weigh integrity, ethics of new technology,” from MLive. Interestingly, a lot of what this story is about is how teachers are using AI to develop assignments, and also to do some things like helping students who don’t necessarily speak English as their native language:

Serving the roughly 30% of [Ypsilanti Community Schools] students who can speak a language other than English, the English Learner Department has found multiple ways to bring AI into the classroom, including helping teachers develop multilingual explanations of core concepts discussed in the curriculum — and save time doing it.

“A lot of that time saving allows us to focus more on giving that important feedback that allows students to grow an be aware of their progress and their learning,” [Teacher Connor] Laporte said.

Laporte uses an example of a Spanish-speaking intern who improved a vocabulary test by double-checking the translations and using ChatGPT to add more vocabulary words and exercises. Another intern then used ChatGPT to make a French version of the same worksheet.

A lot of the theme of this article is about how teachers have moved beyond being terrified of AI ruining everything to becoming a tool to work with in teaching. That’s happening in lots of places and lots of ways; for example, as Inside Higher Ed noted, “Art Schools Get Creative Tackling AI.” It’s a long piece with a somewhat similar theme: not necessarily embracing AI, but also recognizing the need to work with it.

MLA apparently now has “rules” for how to cite AI. I guess maybe it isn’t the end of the essay then, huh? Of course, that doesn’t mean that a lot of writers are going to be happy about AI.  This one is from a while ago, but in The Atlantic back in September, Alex Reisner wrote about “These 183,000 Books are Fueling the Biggest Fight in Publishing and Tech.” Reisner had written earlier about how Meta’s AI systems were being trained on a collection of more than 191,000 books that were often used without permission. The article has a search feature so you can see if your book(s) were a part of that collection. For what it’s worth, my book and co-edited collection about MOOCs did not make the cut.

Several famous people/famous writers are now involved in various lawsuits where the writers are suing the AI companies for using their work without permission to train (“teach?”) the AIs. There’s a part of me that is more than sympathetic to these lawsuits. After all, I never thought it was fair that companies like Turnitin can use student writing without permission as part of its database for detecting plagiarism. Arguably, this is similar.

But on the other hand, OpenAI et al didn’t “copy” passages from Sarah Silverman or Margaret Atwood or my friend Dennis Danvers (he’s in that database!) and then try to present that work as something the AI wrote. Rather, they trained (taught?) the AI by having the program “read” these books. Isn’t that just how learning works? I mean, everything I’ve ever written has been been influenced in direct and indirect ways by other texts I’ve read (or watched, listened to, seen, etc). Other than scale (because I sure as heck have not read 183,000 books), what’s the difference between me “training” by reading the work of others and the AI doing this?

Of course, even with all of this training and the continual tweaking of the software, AIs still have the problem of making shit up. Cade Metz wrote in The New York Times “Chatbots May ‘Hallucinate’ More Often Than Many Realize.” Among other things, the article is about a new start-up called Vectara that is trying to estimate just how often AIs “hallucinate,” and (to leap ahead a bit) they estimated that different AIs hallucinate at different rates ranging from 3% to 27% of the time. But it’s a little more complicated than that.

Because these chatbots can respond to almost any request in an unlimited number of ways, there is no way of definitively determining how often they hallucinate. “You would have to look at all of the world’s information,” said Simon Hughes, the Vectara researcher who led the project.

Dr. Hughes and his team asked these systems to perform a single, straightforward task that is readily verified: Summarize news articles. Even then, the chatbots persistently invented information.

“We gave the system 10 to 20 facts and asked for a summary of those facts,” said Amr Awadallah, the chief executive of Vectara and a former Google executive. “That the system can still introduce errors is a fundamental problem.”

If I’m understanding this correctly, this means that even when you give the AI a fairly small data-set to analyze (10-20 “facts”), the AI still makes shit up with things not a part of that data-set. That’s a problem.

But it still might not stop me from trying to develop some kind of ChatGPT/AI-based grading tool, and that might be about to get a lot easier. (BTW, talk about burying the lede after that headline!)  OpenAI announced something they’re calling (very confusingly) “GPTs,” which (according to this article by Devin Coldewey in TechCrunch) is “a way for anyone to build their own version of the popular conversational AI system. Not only can you make your own GPT for fun or productivity, but you’ll soon be able to publish it on a marketplace they call the GPT Store — and maybe even make a little cash in the process.”

Needless to say, my first thought was could I use this to make an AI Grading tool? And do I have the technical skills?

As far as I can tell from OpenAI’s announcement about this,  GPTs require upgrading to their $20 a month package and it’s just getting started– the GPT store is rolling out later this month, for example.  Kevin Roose of The New York Times has a thoughtful and detailed article about the dangers and potentials of these things, “Personalized A.I. Agents Are Here. Is the World Ready for Them?” User-created agents will very soon be able to automate responses to questions (that OpenAI announcement has examples like a “Creative Writing Coach,” a “Tech Advisor” for trouble-shooting things, and a “Game Time” advisor that can explain the rules of card and board games. Roose writes a fair amount about how this technology could also be used by customer service or human resource offices, and to handle things like responding to emails or updating schedules. Plus none of this requires any actual programming skills, so I am imagining something like “If This Then That” but much more powerful.

AI agents might also be made to do evil things, which has a lot of security people worried for obvious reasons. Though I don’t think these agents are going to be to powerful enough to do anything too terrible; actually, I don’t think these agents will have the capabilities to make the AI grading app I want, at least not yet. Roose got early access to the OpenAI project, and his article has a couple of examples of how he played around with it:

The first custom bot I made was “Day Care Helper,” a tool for responding to questions about my son’s day care. As the sleep-deprived parent of a toddler, I’m always forgetting details — whether we can send a snack with peanuts or not, whether day care is open or closed for certain holidays — and looking everything up in the parent handbook is a pain.

So I uploaded the parent handbook to OpenAI’s GPT creator tool, and in a matter of seconds, I had a chatbot that I could use to easily look up the answers to my questions. It worked impressively well, especially after I changed its instructions to clarify that it was supposed to respond using only information from the handbook, and not make up an answer to questions the handbook didn’t address.

That sounds pretty cool, and I bet I could create an AI agent capable of writing an summative end-comment on a student essay based on a detailed grading rubric I feed into the machine. But that’s a long way from doing the kind of marginal commenting on student essays that responds to particular sentences, phrases, and paragraphs. I want an AI agent/grading tool that can “read” a piece of student writing that is more like how I would read and comment on a piece of student writing, and that  limited to a rubric.

But this is getting a lot closer to being potentially useful– not a substitute for me actually reading and evaluating student writing, but as a tool to make it easier to do. Right now, the free version of ChatGPT does a good job of revising away grammar and style mistakes and errors, so maybe instead of me making marginal comments on a draft about these issues, students can first try using the AI to help them do this kind of low-level revision before they turn it in. That, combined with a detailed end comment from the AI might, actually work well. I’m not quite sure if this would actually save me any time since it seems like setting up the AI to do this would take a lot of time, and I have a feeling I’d have to set up the AI agent for every unique assignment. Plus, and in addition to the time it would take to set up, this would cost me $20 a month.

Maybe for next semester….

Posted on

So, What About AI Now? (A talk and an update)

A couple of weeks ago, I gave a talk/lead a discussion called “So, What About AI Now?” That’s a link to my slides. The talk/discussion was for a faculty development program at Washtenaw Community College, a program organized by my friend, colleague, and former student, Hava Levitt-Phillips.

I covered some of the territory I’ve been writing about here for a while now and I thought both the talk and discussion went well. I think most of the people at this thing (it was over Zoom, so it was a little hard to read the room) had seen enough stories like this one on 60 Minutes the other night: Artificial Intelligence is going to at least be as transformative of a technology as “the internet,” and there is not a zero percent chance that it could end civilization as we know it. All of which is to say we probably need to put the dangers of a few college kids using AI (badly) to cheat on poorly designed assignments into perspective.

I also talked about how we really need to question some of the more dubious claims in the MSM about the powers of AI, such as the article in the Chronicle of Higher Education this past summer, “GPT-4 Can Already Pass Freshman Year at Harvard.”  I blogged about that nonsense a couple months ago here, but the gist of what I wrote there is that all of these claims of AI being able to pass all these tests and freshman year at Harvard (etc.) are wrong. Besides the fact that the way a lot of these tests are run make the claims bogus (and that is definitely the case with this CHE piece), students in our classes still need to show up– and I mean that for both f2f and online courses.

And as we talked about at this session, if a teacher gives students some kind of assignment (an essay, an exam, whatever) that can be successfully completed without ever attending class, then that’s a bad assignment.

So the sense that I got from this group– folks teaching right now the kinds of classes where (according to a lot of the nonsense that’s been in MSM for months) the cheating with ChatGPT et al was going to just make it impossible to assign writing anymore, not in college and not in high school— is it hasn’t been that big of a deal. Sure, a few folks talked about students who tried to cheat with AI who were easily caught, but for the most part it hadn’t been much of a problem. The faculty in this group seemed more interested in trying to figure out a way to make use of AI in their teaching than they were in cheating.

I’m not trying to suggest there’s no reason to worry about what AI means for the future of… well, everything, including education. Any of us who are “knowledge workers”– that is, teachers, professors, lawyers, scientists, doctors, accountants, etc. etc.– needs to pay attention to AI because there’s no question this shit is going to change the way we do our jobs. But my sense from this group (and just the general vibe I get on campus and in social media) is that the freak-out about AI is over, which is good.

One last thing though:  just the other day (long after this talk), I saw what I believe to be my first case of a student trying to cheat with ChatGPT– sort of. I don’t want to go into too many details since this is a student in one of my classes right now. But basically, this student (who is struggling quite a bit) turned in a piece of writing that was first and foremost not the assignment I gave, and it also just happened this person used ChatGPT to generate a lot of the text. So as we met to talk about what the actual assignment was and how this student needed to do it again, etc., I also started asking about what they turned in.

“Did you actually write this?” I asked. “This kind of seems like ChatGPT or something.”

“Well, I did use it for some of it, yes.”

“But you didn’t actually read this book ChatGPT is citing here, did you?”

“Well, no…”

And so forth.  Once again, a good reminder that students who resort to cheating with things like AI are far from criminal masterminds.

Posted on

No, an AI could not pass “freshman year” in college

I am fond of the phrase/quote/mantra/cliché “Ninety percent of success in life is just showing up,” which is usually attributed to Woody Allen. I don’t know if Woody was “the first” person to make this observation (probably not, and I’d prefer if it was someone else), but in my experience, this is very true.

This is why AIs can’t actually pass a college course or their freshmen year or law school or whatever: they can’t show up. And it’s going to stay that way, at least until we’re dealing with advanced AI robots.

This is on my mind because my friend and colleague in the field, Seth Kahn, posted the other day on Facebook about this recent article from The Chronicle of Higher Education by Maya Bodnick, “GPT-4 Can Already Pass Freshman Year at Harvard.” (Bodnick is an undergraduate student at Harvard). It is yet another piece claiming that the AI is smart enough to do just fine on its own at one of the most prestigious universities in the world.

I agreed with all the other comments I saw on Seth’s post. In my comment (which I wrote before I actually read this CHE article), I repeated three points I’ve written about here or on social media before. First, ChatGPT and similar AIs can’t evaluate and cite academic research at even the modest levels I expect in a first year writing class. Second, while OpenAI proudly lists all the “simulated exams” where ChatGPT has excelled (LSAT, SAT, GRE, AP Art History, etc.), you have to click the “show more exams” button on that page to see that none of the versions of their AI has managed better than a “2” on the AP English Language (and also Literature) and Composition exams. It takes a “3” on this exam to get any credit at EMU, and probably a “4” at a lot of other universities.

Third, I think mainstream media and all the rest of us really need to question these claims of AIs passing whatever tests and classes and whatnot much MUCH more carefully than I think most of us have to date.  What I was thinking about when I made that last comment was another article published in CHE and in early July, “A Study Found That AI Could Ace MIT. Three MIT Students Beg to Differ,” by Tom Bartlett. In this article, Bartlett discusses  a study (which I don’t completely understand because it’s too much math and details) conducted by 3 MIT students (class of 2024) who researched the claim that an AI could “ace” MIT classes. The students determined this was bullshit. What were the students’ findings (at least the ones I could understand)? In some of the classes where the AI supposedly had a perfect score, the exams include unsolvable problems, so it’s not even possible to get a perfect score. In other examples, the exam questions the AI supposedly answered correctly did not provide enough information for that to be possible either. The students posted their results online and at least some of the MIT professors who originally made the claims agreed and backtracked.

But then I read this Bodnick article, and holy-moly, this is even more bullshitty than I originally thought. Let me quote at length Bodnick describing her “methodology”:

Three weeks ago, I asked seven Harvard professors and teaching assistants to grade essays written by GPT-4 in response to a prompt assigned in their class. Most of these essays were major assignments which counted for about one-quarter to one-third of students’ grades in the class. (I’ve listed the professors or preceptors for all of these classes, but some of the essays were graded by TAs.)

Here are the prompts with links to the essays, the names of instructors, and the grades each essay received:

  • Microeconomics and Macroeconomics (Jason Furman and David Laibson): Explain an economic concept creatively. (300-500 words for Micro and 800-1000 for Macro). Grade: A-
  • Latin American Politics (Steven Levitsky): What has caused the many presidential crises in Latin America in recent decades? (5-7 pages) Grade: B-
  • The American Presidency (Roger Porter): Pick a modern president and identify his three greatest successes and three greatest failures. (6-8 pages) Grade: A
  • Conflict Resolution (Daniel Shapiro): Describe a conflict in your life and give recommendations for how to negotiate it. (7-9 pages). Grade: A
  • Intermediate Spanish (Adriana Gutiérrez): Write a letter to activist Rigoberta Menchú. (550-600 words) Grade: B
  • Freshman Seminar on Proust (Virginie Greene): Close read a passage from In Search of Lost Time. (3-4 pages) Grade: Pass

I told these instructors that each essay might have been written by me or the AI in order to minimize response bias, although in fact they were all written by GPT-4, the recently updated version of the chatbot from OpenAI.

In order to generate these essays, I inputted the prompts (which were much more detailed than the summaries above) word for word into GPT-4. I submitted exactly the text GPT-4 produced, except that I asked the AI to expand on a couple of its ideas and sequenced its responses in order to meet the word count (GPT-4 only writes about 750 words at a time). Finally, I told the professors and TAs to grade these essays normally, except to ignore citations, which I didn’t include.

Not only can GPT-4 pass a typical social science and humanities-focused freshman year at Harvard, but it can get pretty good grades. As shown in the list above, GPT-4 got all A’s and B’s and one Pass.

JFC. Okay, let’s just think about this for a second:

  • We’re talking about three “essays” that are less than 1000 words and another three that are slightly longer, and based on this work alone, GPT-4 “passed” a year of college at Harvard. That’s all it takes. Really; really?! That’s it?
  • I would like to know more about what Bodnick means when she says that the writing prompts were “much more detailed than the summaries above” because those details matter a lot. But as summarized, these are terrible assignments. They aren’t connected with the context of the class or anything else.  It would be easy to try to answer any of these questions with a minimal amount of Google searching and making educated guesses. I might be going out on a limb here, but I don’t think most writing assignments at Harvard or any other college– even badly assigned ones– are as simplistic as these.
  • It wasn’t just ChatGPT: she had to do some significant editing to put together ChatGPT’s short responses into longer essays. I don’t think the AI could have done that on its own. Unless it hired a tutor.
  • Asking instructors to not pay any attention to the lack of citation (and I am going to guess the need for sources to back up claims in the writing) is giving the AI way WAAAAYYY too much credit, especially since ChatGPT (and other AIs) usually make shit up hallucinate when citing evidence. I’m going to guess that even at Harvard, handing in hallucinations would result in a failing grade. And if the assignment required properly cited sources and the student didn’t do that, then that student would also probably fail.
  • It’s interesting (and Bodnick points this out too) that the texts that received the lowest grades are ones that ask students to “analyze” or to provide their opinions/thoughts, as opposed to assignments that were asking for an “information dump.” Again, I’m going to guess that, even at Harvard, there is a higher value placed on students demonstrating with their writing that they thought about something.

I could go on, but you get the idea. This article is nonsense. It proves literally nothing.

But I also want to return to where I started, the idea that a lot of what it means to succeed in anything (perhaps especially education) is showing up and doing the work. Because after what seems like the zillionth click-bait headline about how ChatGPT could graduate from college or be a lawyer or whatever because it passed a test (supposedly), it finally dawned on me what has been bothering me the most about these kinds of articles: that’s just not how it works! To be a college graduate or a lawyer or damn near anything else takes more than passing a test; it takes the work of showing up.

Granted, there has been a lot more interest and willingness in the last few decades to consider “life experience” credit as part of degrees, and some of these places are kind of legitimate institutions– Southern New Hampshire and the University of Phoenix immediately come to mind. But “life experience” credit is still considered mostly bullshit and the approach taken by a whole lot of diploma mills, and real online universities (like SNHU and Phoenix) still require students to mostly take actual courses, and that requires doing more than writing a couple papers and/or taking a couple of tests.

And sure, it is possible to become a lawyer in California, Vermont, Virginia and Washington without a law degree, and it is also possible to become a lawyer in New York or Maine with just a couple years of law school or an internship. But even these states still require some kind of experience with a law office, most states do require attorneys to have law degrees, and it’s not exactly easy to pass the bar without the experience you get from earning a law degree. Ask Kim Kardashian. 

Bodnick did not ask any of the faculty who evaluated her AI writing examples if it would be possible for a student to pass that professor’s class based solely on this writing sample because she already knew the answer: of course not.

Part of the grade in the courses I teach is based on attendance, participation in the class discussions and peer review, short responses to readings, and so forth. I think this is pretty standard– at least in the humanities. So if some eager ChatGPT enthusiast came to one of my classes– especially one like first year writing, where I post all of the assignments at the beginning of the semester (mainly because I’ve taught this course at least 100 times at this point)– and said to me “Hey Krause, I finished and handed in all the assignments! Does that mean I get an A and go home now?” Um, NO! THAT IS NOT HOW IT WORKS! And of course anyone familiar with how school works knows this.

Oh, and before anyone says “yeah, but what about in an online class?” Same thing! Most of the folks I know who teach online have a structure where students have to regularly participate and interact with assignments, discussions, and so forth. My attendance and participation policies for online courses are only slightly different from my f2f courses.

So please, CHE and MSM in general: stop. Just stop. ChatGPT can (sort of) pass a lot of tests and classes (with A LOT of prompting from the researchers who really really want ChatGPT to pass), but until that AI robot walks/rolls into  a class or sets up its profile on Canvas all on its own, it can’t go to college.