Anti-Plagiarism Service Turnitin Is Building a Tool To Detect ChatGPT-Written Essays
Turnitin’s VP of AI, Eric Wang, said there are obvious patterns in AI writing that computers can detect. “Even though it feels human-like to us, [machines write using] a fundamentally different mechanism. It’s picking the most probable word in the most probable location, and that’s a very different way of constructing language [compared] to you and I,” he told The Register. […] ChatGPT, however, doesn’t have this kind of flexibility and can only generate new words based on previous sentences, he explained. Turnitin’s detector works by predicting what words AI is more likely to generate in a given text snippet. “It’s very bland statistically. Humans don’t tend to consistently use a high probability word in high probability places, but GPT-3 does so our detector really cues in on that,” he said.
Wang said Turnitin’s detector is based on the same architecture as GPT-3 and described it as a miniature version of the model. “We are in many ways I would [say] fighting fire with fire. There’s a detector component attached to it instead of a generate component. So what it’s doing is it’s reading language in the exact same way GPT-3 reads language, but instead of spitting out more language, it gives us a prediction of whether we think this passage looks like [it’s from] GPT-3.” The company is still deciding how best to present its detector’s results to teachers using the tool. “It’s a difficult challenge. How do you tell an instructor in a small amount of space what they want to see?” Chechitelli said. They might want to see a percentage that shows how much of an essay seems to be AI-written, or they might want confidence levels showing whether the detector’s prediction confidence is low, medium, or high to assess accuracy. “I think there is a major shift in the way we create content and the way we work,” Wang added. “Certainly that extends to the way we learn. We need to be thinking long term about how we teach. How do we learn in a world where this technology exists? I think there is no putting the genie back in the bottle. Any tool that gives visibility to the use of these technologies is going to be valuable because those are the foundational building blocks of trust and transparency.”
Read more of this story at Slashdot.