AI Is Good for for Production, Bad for Learning

The faster AI helps us finish, the less we learn from the process

I find myself using AI more and more often. It has helped me to both produce more stuff and more different kinds of things than I would have been able to do otherwise. But I’ve also started to notice that the more I let AI do my thinking for me, the less I seem to retain myself.

Let’s start with the good. In my work life, AI can be a great data organizer and copy editor. I often use it to suggest title ideas. And I had it scrape a website for data that would have taken me days to capture by hand—in the olden days, I probably would have decided it was impossible and given up.

For Read Not Guess, AI is built into the app version and allows me to quickly create audio files and thematic pictures. Separately, I used it to help me create a cover for a workbook that I wrote.

But AI is not all good. I’ve also had AI summarize some policy documents and research papers for me, and I’ve come to regret those instances. Why? Because that’s where the learning happens. And to be good at my job, I need to know stuff. Information is freely available, now more so than ever. The information itself is not what’s valuable; what’s important is the ability be able to recall and understand and combine complex pieces of information.

In fact, I should have known better. One of the first pieces of advice I give to young people starting out in this field is to go to the primary sources. I tell them not to rely on the NY Times or EdWeek summary and instead read the research study or piece of legislation themselves. I find I get a deeper understanding of topics when I go to the primary source, and I can then (mentally) call B.S. when someone doesn’t truly understand the limits or implications. So I understand—and preach!—the importance of going to the source. And yet here I was breaking my own rule thanks to the convenience of AI tools.

AI is good for speed, bad for learning

A recent study from a team of researchers led by Sina Rismanchian at the University of California-Irvine dug into this question in the education world. Called, “Faster Completion, Less Learning: Generative AI Reduced Study Time on Math Problems and the Knowledge They Build,” it confirms what I’ve been feeling in my own life.

It’s a clever study. It uses ten years of data from ALEKS, which the paper describes as “a widely deployed adaptive mathematics learning and assessment platform serving more than four million students annually.” The researchers compare two different types of math problems on the platform, text-based word problems versus interactive graphing problems. The word problems are susceptible to AI, while the graphing problems are not.

The authors find a steep drop-off in the amount of time that students spend on the word problems after the release of ChatGPT, especially for high school and college students. In fact, time spent on word problems, “declines 2.8% per quarter among college students after ChatGPT’s release, cumulating to 26.9% over eleven quarters.” High schoolers show an even bigger decline, whereas there wasn’t much decline for middle schoolers and no decline for 5th graders.

Are students just getting better at word problems? Probably not.

The authors show that the time effects completely disappear in settings where the math questions are proctored (observed) and AI use is not allowed. Left to their own devices—pun intended—students are resorting more and more often to time-saving methods.

The study can't observe students using AI directly. But the timing is hard to ignore, and there’s clearly some magic going on behind the screen that can’t explain the sudden speed with which students are able to complete math word problems, and why that speed only shows up in unsupervised settings.

But maybe the students are still learning the material? This study can’t answer that question directly, but it appears to be no.

Again turning back to the proctored versus non-proctored formats, students with supervision who were not able to use AI tools had a 25% lower chance of getting a given word problem correct. On the other side, students suddenly got a lot more problems correct in non-proctored environments.

The fact that these are pointing in opposite directions is a real problem for schools. That is, students might appear to get through more homework and be completing their at-home assignments accurately, but the only way to know for sure is to measure learning in person, without the aid of AI. Recently, a professor at Brown ran this exact experiment and saw student grades plummet when they were not able to draw on AI to complete their work.

In other words, AI creates the illusion of learning. More work gets done. Homework is easier to complete. But when students are asked to demonstrate what they actually know without AI, the gains may disappear.

---

Reading List

Jorge Elorza: The interests of teachers unions and students are not always aligned

Anna Stokke: Five Math Education Misconceptions That Are Holding Students Back