Published: Fri 10 July 2026
By Joseph Reagle
In praxis .
tags: teaching ai
As the fall semester approaches I have two questions. How good is Pangram.com at avoiding false positives
for AI prose, and has student AI use grown since ChatGPT’s arrival?
Fortunately, I have a dataset with which to answer these questions:
seven years of reading responses from my Communication
in the Digital Age course.
Unfortunately, I don’t have all the students’ responses over that
time, but many of their responses do survive on the students’ HackMD
wiki pages. I crawled those pages and sampled about fifty responses per
semester. I’d test them all, except this sample itself cost me around
$30 in Pangram API costs.
Pangram rates each of the 278 responses from before ChatGPT’s
November 2022 launch as human-written. That’s reassuring with respect to
avoiding false positives.
Pangram’s flagged-AI-or-Mixed rate by term (578 of 2,354
responses sampled)
Term
Period
Total
Sampled
AI detected
2019–2022 Spring
pre
1245
228
0.0%
2022 Fall
transition
177
50
0.0%
2023 Spring
post
171
50
6.0%
2023 Fall
post
128
50
6.0%
2024 Spring
post
172
50
26.0%
2025 Spring
post
190
50
18.0%
2025 Fall
post
114
50
42.0%
2026 Spring
post
157
50
6.0%
After ChatGPT’s introduction in November 2022, there’s an upward
trend in detected AI prose, climbing to 42% in Fall 2025. Curiously, the
Spring 2026 drops back to 6%. That was the semester I ran Pangram on
all readings responses, told students I was doing so, gave
zeros to the responses that were written by AI, and referred students
who lied about it to the Office of Student Conduct
and Conflict Resolution . (I heard some real whoppers this past
year.) As I wrote about my process earlier, I also required a link
to writings’ version histories and reviewed drafts’ histories with
the Process Feedback
tool.
So did enforcement work, or did it make students better at hiding it?
I don’t know. Stupid
misuse of AI is detectable, but others might be getting more sneaky .
My policy and its enforcement does have two costs I can detect
elsewhere. This past year, my courses saw an unprecedented number of
students dropping in the first week—once I explained the AI misuse
policy and its enforcement. The spring semester also yielded the lowest
student teaching
evaluations I’ve ever received in the 2019–2026 period. I think the
two are related: it’s too tempting for students to misuse AI on
out-of-class writing, want to avoid courses that are rigorously policed,
and resent the detection and enforcement that follows.
This isn’t tenable. Though it pains me, I will move further from
writing towards testing. In Fall 2027, I’ll abandon the requirement for
well-written reading responses and ask students instead to share
small-stakes QICs (Question, Insight, or Connection) from the readings.
I will adopt higher-stakes but easy in-class quizzes, perhaps counting
the best seven out of ten for that grade. In the quizzes, however, I
will sometimes ask students to document the QIC they shared before
class. If they did the work—or even pay attention to what AI told
them—they should be able to answer this and participate in class.
There are comments .