AI in Education Has Its Limits

While most students seem to be really excited to use AI in education, some teachers have mixed feelings. I've taught for a number of years, and I still do quite a bit of work with AI. When my daughter gets stuck on her math homework, and I can't figure out the answer myself, we usually ask ChatGPT for help. I use ChatGPT 4.0, and it generally gives us what we need. 

If we ask for help setting up a problem, it can do that. If we're working on a problem and get stuck, I can take a photograph of our work, upload it to ChatGPT, and it can tell us where we went wrong. We could, of course, ask it to solve the problem for us outright, but that's not exactly helpful when it comes to test time, so we usually use it as an at-home teacher, and it works great for that.

I don't teach in the classroom anymore, but I'm still involved in education on a number of fronts. A few months ago, I worked with a former colleague, thinking that it would be pretty easy to have ChatGPT generate quizzes and grade those quizzes. 

With this approach, we could differentiate the material seamlessly, creating different quizzes for different students based on where they were academically. Coordination (joining two independent clauses with a coordinating conjunction or conjunctive adverb) and subordination (joining two independent clauses with a subordinating conjunction) seemed like a pretty straightforward exercise, but it was immediately a disaster. 

ChatGPT can't get the hang of this very simple thing. I keep trying, trying to teach ChatGPT how to work through these problems, but it simply can't. 

Before I sat down to type this, I ran through a series of exercises with ChatGPT 4.0, trying to get it to make a five-question quiz on coordination. ChatGPT 4.0 is the latest build for which I have access:

Chatty (which is what I’ve named my AI) knocked it out of the park with this one. This is a good and legitimate test, and I was quickly able to work through each problem and arrive at a correct answer.

I was expecting this, though. Chatty always does great with true and false stuff (for this topic). 

The real test was going to be whether Chatty could construct a multiple-choice test. Most teachers will tell you that multiple-choice quizzes are the best way to figure out if a kid really knows something or not.

So far so good. The first two looked solid (and the answers are C and B). Question number 3, though …

I immediately knew where Chatty was going here – she was going to say that C was the correct answer, but it’s not. As she said in the question 4 response for the true/false test, each clause must be able to stand alone as a complete sentence. "She likes to draw" is a complete sentence, but "paint" is not. For C to be correct, it would have to be "She likes to draw, and she likes to paint." 

This is a pretty basic thing, and plenty of children are able to do this kind of work by themselves daily. It's particularly frustrating because ChatGPT's own logic knows the rules to this, but it can't seem to put those rules into practice for this one particular topic. 

I've worked with ChatGPT quite a bit, but it never seems to learn. It's not like the definition of a clause is ambiguous, either. It's simply a collection of words that has both an identifiable subject and a complete verb, forming a complete thought. Anyway, I decided to give Chatty some guidance. I answered “none” for the broken question:

I asked Chatty to justify her reasoning for question 3:

She didn’t come out and admit to being wrong, but she sort of conceded the point. I wasn’t going to let it go, though:

My point with this basic example is that AI is not quite there yet. Not in regard to this aspect of education, at least. From what I can see, ChatGPT cannot successfully teach or even solve problems related to clause coordination. 

There's definitely a place for AI in education, and AI is getting better every day. But I've been working with AI for the last three or four months on this one specific thing, and it's never been able to crack that nut.