By Jeff Schwartz, Instructional Designer; Stephen Riley, Director of Academic Assessment
Updated May 10, 2024
Almost immediately after the large-scale introduction of the Artificial Intelligence Large Language Models like ChatGPT, tech companies (and at least one student) began developing software to counteract the program’s misuses. For the last year now, Turnitin, a plagiarism prevention service that is integrated with Canvas’s SpeedGrader, has included an “AI Writing Detector,” which assigns a score out of 100 meant to indicate the probability that student work has been generated by an AI writing program.
The goals of Turnitin, and its AI checker more specifically, are obviously noble and intended to help faculty identify and discourage academic dishonesty. But no tool does everything. To put it bluntly, Turnitin’s AI checker is no panacea for weeding out AI-generated texts. This is a tool that should be used with caution and in tandem with other practices. Indeed, any use of Turnitin should proceed with caution, due diligence, and common sense.
What is Turnitin, anyway?
We will return to how Turnitin’s AI checker works, and advice on how best to use it, but first we thought it would be beneficial to say a bit about the other features of Turnitin.
Even according to its own website, Turnitin is not a plagiarism detector. Instead, it is better to think of it, in the words of one study, as a “‘text-matching tool’ that electronically checks the similar text of submitted material against web contents, its database of academic publication and earlier submitted documents.”
If you choose to enable Turnitin, you will see a percentage score out of 100 when you pull up an assignment in SpeedGrader; the lower the score, the less likely (according to Turnitin, at least) that the text bears any similarity to other texts in Turnitin’s database.
It should be said as well that this “similarity score” does not factor in Turnitin’s AI Writing Detector (which is discussed in more detail below). Thus, even a low similarity score does not necessarily mean an assignment was generated without the use of AI.
What to know about using, or not using, Turnitin
Two of the best things about Turnitin are that it is optional and customizable.
To the first point, if you do not think your course would benefit from using Turnitin, you can choose not to use it. In the settings for an assignment, simply make sure under the menu for “Plagiarism Review” you have selected “none.”
Deciding not to use Turnitin is a perfectly valid decision; text-matching tools are not beneficial across disciplines and may not comport with your pedagogical approach within the classroom.
If, however, you want to use Turnitin to determine the similarity of student texts to other texts – or better yet (and more on this below) you want to use it as a teaching tool within the classroom – you can select Turnitin from the Plagiarism Review menu. Then, you can decide what features you want to utilize. For example, Turnitin can theoretically exclude quotes or bibliographic material from its similarity report if you check the appropriate boxes.
We say “theoretically” because this feature is not foolproof and will sometimes include properly quoted and cited material in its similarity report.
Another feature is the Student Repository, which is a digital storehouse of student work submitted through Turnitin. If you enable this feature, Turnitin will compare a student’s paper against any other work submitted to Turnitin. And, as with the “exclude” settings outlined above, this feature is also not foolproof. To test it out, we used a paper that a student had turned in for an assignment in one of our courses and uploaded the paper as a “Test Student” in Canvas. The similarity score generated for the Test Student was 97% (and color-coded a forbidding red). Obviously, 97% betrays more certainty than, say, 50%. But it is interesting, and useful to know, that even in cases without ambiguity, Turnitin cannot be depended upon for 100% certainty (a fact that will become even more important when we discuss its AI checker below).
Once students turn in work, you can access their similarity score in SpeedGrader.
Clicking on the score will bring up a new tab that allows you to view the sections of a text that Turnitin has marked as similar to other texts. You can also choose if students can see their similarity scores. Students can view their scores by navigating to an assignment, and selecting “submission details.”
Letting students see their similarity score can be a worthwhile practice if an instructor has concerns and wants to share those concerns with students. Even better, the similarity score could be used as a formative assessment practice. For example, say you ask students to submit a draft of an assignment for instructor and/or peer feedback. If you also allowed students to view their similarity score, they could use their score to check and refine their work.
More broadly, using Turnitin could occasion an opportunity to discuss issues of honesty, academic integrity, and the uses and limitations of technology with your students.
Turnitin’s AI Writing Indicator
And speaking of the uses and limitations of technology, let’s turn, as it were, to Turnitin’s latest feature: its AI Writing Detector.
According to Turnitin’s website, the AI Writing Detector works by cross-checking segments of text (usually a few hundred words) against its proprietary AI detection algorithm, and awards the text either a zero or a one; zero means the text was likely not produced by AI, while a one means that Turnitin has “98% confidence” the text was produced by AI. The detection algorithm is trained on GPT-3.5, which is the latest free version of ChatGPT (the more advanced GPT-4 is only available to subscribers).
To access the AI Detector’s Report, you’ll first need to click on the Similarity Score in SpeedGrader, and then select the AI Writing tab along the top. Turnitin recently changed its interface, so if you prefer the “classic” look you can toggle over to that mode using the “go to classic view” button on the left.
Despite Turnitin’s sophisticated programming and “98% confidence” in its accuracy, the program is not perfect. For one, it cannot provide scores for documents with less than 300 words or more than 15,000 words. More importantly, however, it can generate false positives.
On its website, Turnitin claims that its false positive rate is “less than 1% for documents with 20% or more AI writing,” which is a change from their previous claim of a less than 1% false positive rate. Furthermore, they go on to say that “there is a 4% likelihood that a specific sentence highlighted as AI-written might be human-written.” Our own tests within the OTL have confirmed this issue. We have uploaded texts written without the use of AI, only to have those texts flagged as containing AI-generated writing.
What can be especially confusing is the way Turnitin displays this information. Below is a screenshot from a false positive instance:
Here the AI Writing Detector is saying that it’s possible (presumably 98% possible, according to Turnitin’s claims of accuracy) that 18% of this text was written by AI. But since this is a “low score” it could be a false positive. There is also a disclaimer at the bottom of every AI score that states: “Our AI writing assessment may not always be accurate (it may misidentify both human and AI-generated text) so it should not be used as the sole basis for adverse actions against a student.”
These conflicting messages make it especially difficult for professors to navigate how much, or even whether, to trust such evaluations.
What’s more, unlike Turnitin’s other features, such as the Similarity Report, there is no further data to justify its AI scores; we simply have to trust that its algorithm is accurate.
To sum up, Turnitin is 98% confident in its AI writing detection software…but not so confident that it does not feel the need to add warnings about trusting its own AI score.
All of which begs the question, what should you do if you suspect a student has used AI to complete an assignment?
Recommendations for if you suspect plagiarism
As we recommended in a prior blog post, whether you use Turnitin or not, if you suspect a student of using AI to complete an assignment, “Instead of assuming the worst and demonizing the student, take the time to have a conversation.” For instance, ask the student to come to office hours. Perhaps open the discussion by asking the student to talk about their process, how they arrived at certain ideas or located sources. If a student cannot capably or credibly talk about how they created their work, it is possible (though not certain) they may have used AI. And if you suspect more than one student of having used AI to complete an assignment, it could be beneficial for the class to have a broader discussion about the appropriate and inappropriate uses of AI, research, academic ethics, and DU’s honor code.
Below we discuss some recommended practices for promoting academic integrity and dealing with misconduct:
Have a discussion
- Review DU’s honor code and discuss the importance of academic integrity with your students.
- More information about Academic Integrity can be found at the Student’s Rights and Responsibility website and at this Office of Teaching and Learning resource.
- Make your expectations regarding student use of AI explicit in your syllabus and discuss this issue with students on the first day of class. Be specific and give both positive and negative examples for students to understand your expectations.
- Regularly remind students of the importance of academic integrity and your expectations regarding the use of AI when discussing assignments.
- This Colin and Samir podcast includes a helpful discussion about how students could think about their creativity in light of the copy and paste culture of YouTube.
Plan ahead
- Create assignments that lead to authentic assessment, where students must demonstrate their own mastery of course outcomes in relationship to the important problems of the world. For more information, consider the following articles:
- Pietryka, M.T.; Glazier, A. Learning through Collaborative Data Projects: Engaging Students and Building Rapport. Educ. Sci. 2022, 12, 897.
- Supiano, Beckie. Will ChatGPT Change How Professors Assess Learning? It won’t be easy without their colleges’ support. The Chronicle of Higher Education. 2023, 4.
- Consider the following ideas for assignments:
- Draft prompts that reference, engage with, and/or build on other class assignments.
- Consider incorporating personal reflections from students into essays where possible.
- Require at least some writing to be done in class (by hand if possible).
- Consider requiring students to submit multiple drafts of an essay (using what is often referred to as a “scaffolding” or “chunk” approach to writing instruction) that students then revise based upon feedback.
- Ask students to use diverse media, such as a video or audio file, that goes along with or in place of a writing assignment.
- Extend Flipped Learning in your classroom. Ask students to read, view, and digest material at home, and then apply, demonstrate, and perform in class.
- Consider encouraging a specific use of AI into an assignment.
Assess
- Consider incorporating a verbal exam requirement tied to an essay so that students must discuss what they are submitting.
- Require a combination of in-person and outside-of-class writing assignments so that a student’s outside-of-class writing can be compared to their writing produced in class.
- Look for some of the marks of plagiarized or artificial writing:
- Vague general comparisons that tend to “beg the question” as opposed to getting at the heart of an essay topic.
- Factual claims that are asserted confidently but turn out to be false on closer inspection.
- Claims that are backed up with citations of “evidence” but evidence that does not support the claims made.
- Lack of connections between and among ideas, or a lack of substantive analysis.
Parting thoughts
There is no perfect tool to help you determine if a student has plagiarized or improperly used AI to complete an assignment. While Turnitin does have capabilities to check for such things, it is limited and must be used with a significant dose of critical reflection.
Another important thing to bear in mind is the proverbial arms race between AI and AI checkers. Turnitin will no doubt continue to iterate on its programming and improve its ability to detect AI. At the same time, AI will continue to grow more sophisticated. All of which means it is unlikely there will ever be a foolproof AI checker that you can invest 100% confidence in.
It is also clear that policing students’ academic integrity will require significant time and effort that not everyone has the ability to follow through on. The best approach will be a combination of the practices mentioned above.
In the meantime, keep an eye on our website and calendar for more resources and events related to AI. And if you would benefit from a one-on-one consultation, feel free to schedule an appointment with an OTL Instructional Designer.
