I don't think 100% accuracy is logically possible. Because it's entirely possible that someone would just naturally write the exact same thing as an LLM would write. And after the fact there is no way to distinguish the two. But pangram does have an extremely low false positive rate, which I think does make it useful for detecting cheating students. Assuming the base rate of cheating students is 1%, and assuming pangram has a false positive rate of 1 in 10,000 and a true positive rate of 7,000 in 10,000, that means ~98% of students flagged by pangram actually cheated. Combined with a teacher's familiarity with that student's previous work, which should rule out many more false positives, it should be a very useful tool.
> ~98% of students flagged by pangram actually cheated
That 2% is a large number! Of people who will have their integrity impugned for no good reason! That's not okay!
Your calculations also are mixing assignments and students. The rate of false positives of 1/10k is of corpuses, not students. 10k students might each submit 2-3 written assignments per week. Obviously, this greatly increases the impact of the false positive rate.
And all of these numbers are dependent on lab conditions for usage, which are not the case in the real world.
> I don't think 100% accuracy is logically possible.
Yes. Which is why marketing this product as it currently is, is a deeply irresponsible endeavor.
