How AI detectors work
An AI detector does not find a watermark or look anything up. It estimates how typical a passage is of machine-generated text and returns a probability. This guide explains what goes into that estimate and why it is never a certainty.
The basic idea
A language model writes by repeatedly choosing a likely next word. Because it favours likely choices, the text it produces tends to be predictable and evenly paced. Detectors try to measure that. They do not know how a particular piece of text was made. They measure how closely it resembles the patterns of generated text, and the closer it resembles them, the higher the score.
Two families of approach
Signal-based detectors measure properties of the text directly. Two classic ones are predictability (how surprising each word is to a language model, often called perplexity) and burstiness (how much sentence length and structure vary). Simple counts, such as how often stock transitions or repeated phrases appear, sit in the same family.
Trained classifiers take a different route. A model is trained on large numbers of labelled examples of human and machine writing and learns to output a probability. Commercial tools may combine approaches. Vendors generally do not publish exactly what their systems measure, so we cannot tell you what any given product does inside.
Litimus is signal-based. It scores six measurable signals and combines them with fixed weights, and the report shows you the readings. It is deliberately simple and transparent. That is a trade-off: it cannot see patterns that only a large trained model would pick up, and we do not claim it is more accurate than one.
The six signals Litimus uses
- Sentence rhythm (burstiness). The spread of sentence lengths relative to their average. Very even sentences push the score up.
- Stock transitions. Connective phrases such as “moreover” and “it is important to note” from a fixed list, counted against the length of the passage.
- Vocabulary spread. Distinct words as a share of all words. A low share suggests repeated, safe word choices.
- Phrase repetition. Three-word sequences that occur more than once.
- Punctuation habits. Currently the frequency of em dashes.
- Structural symmetry. How similar sentence lengths are, measured in characters.
The weights favour sentence rhythm most, then stock transitions and vocabulary spread. The result is a score from 2 to 98. A score of 66 or above is labelled AI-likely, 34 or below human-likely, and anything between them mixed / unclear.
Why the result is a probability
Human and machine text overlap. A careful writer can produce even, plain prose, and generated text that a human has rewritten can look entirely natural. Any threshold you draw will misplace some of each. So a good detector gives a score and a band, not a yes or no, and says that the middle of the range is inconclusive.
What makes detection harder
- Short text. There is little to measure. OpenAI’s own withdrawn classifier was unreliable on text under 1,000 characters, and Litimus flags anything under 25 words as too short for a stable read.
- Editing. OpenAI noted that AI-written text can be edited to evade detection, and that predictable, formulaic writing is hard to classify either way.
- Other languages. OpenAI’s classifier was reliable only in English, and Litimus is tuned for English.
- Writing style. Independent research found that widely used detectors misclassified non-native English writing as AI-generated, possibly because a more constrained vocabulary and phrasing resembles the patterns detectors look for.
What no detector can do
No detector can prove who wrote something, identify which tool produced it, or tell you whether AI help was allowed. It can only say how much a passage resembles generated text. That is why Litimus’s Terms of Service ask you not to treat a score as proof.
A cautionary example
In early 2023 OpenAI released a classifier for AI-written text and reported that it correctly identified 26% of AI-written text while wrongly labelling 9% of human-written text as AI-written. On 20 July 2023 it withdrew the tool, saying it was no longer available due to its low rate of accuracy. The detectors on the market today are different products, but the episode is a useful reminder to treat any score with caution.