So far, AI companies have struggled to develop tools that reliably detect text generated by large language models (LLMs). Now, a group of researchers has devised a novel method to estimate LLM usage in scientific writing by identifying “excess words” that became much more frequent during the LLM era (2023-2024).
Their findings suggest that “at least 10% of 2024 abstracts were processed with LLMs. That’s equally depressing in a way, as it means that even scientists can’t be bothered to write their own work.
The short version is if you stop using these words, you’re going to be in better shape with the detectors.
- Delves
- Showcasing
- Underscores
- Potential
- Findings
- Crucial
Surprisingly Elevate and ‘In a world of’, didn’t get a mention. But you know about them anyway…
Excess words are also a giveaway and we know that ChatGPT can end up talking in circles with too much passive language. So it pays to edit those parts right out your text if you want to slip past the detectors.
Detectors Suck Now, But Not Forever
AI detectors are a bad joke right now and everything from the Declaration of Independence to 1980s articles are getting flagged as AI. That is causing a lot of trouble for human copywriters that suddenly have to prove their integrity when Quillbot or A N Other says their handwritten prose is purest ChatGPT.
They will get much more reliable, so if you’re planning on being around in a year’s time you might want to take the safe route. Heavily edit your content, or use the likes of Koala AI for a more human voice and then edit that as much as you can. Surfer AI also shows good results with the AI detectors. But if you want to be sure then there’s no substitute for a bit of human effort.
In a pre-print paper on Cornell University’s site, researchers from Germany’s University of Tubingen and Northwestern University drew inspiration from studies measuring the impact of the COVID-19 pandemic by looking at excess deaths. By examining “excess word usage” post-LLM introduction in late 2022, they discovered that “the appearance of LLMs led to an abrupt increase in the frequency of certain style words” that was “unprecedented in both quality and quantity.”
To measure these changes, the researchers analyzed 14 million paper abstracts on PubMed from 2010 to 2024, tracking each word’s yearly frequency. Comparing the expected frequency (based on pre-2023 trends) with the actual frequency in 2023 and 2024, they found that certain words became significantly more common after LLMs were introduced.
For instance, “delves” appeared 25 times more in 2024 papers than expected; “showcasing” and “underscores” saw ninefold increases. Other words like “potential,” “findings,” and “crucial” also became notably more frequent.
Are LLMs an Epidemic?
While language naturally evolves, the researchers noted that such rapid increases were previously only associated with major health events (e.g., “ebola,” “zika,” “coronavirus”). Post-LLM, however, hundreds of words with no such links saw sudden spikes, primarily “style words” like verbs, adjectives, and adverbs (e.g., “additionally,” “comprehensive,” “insights”).
These findings suggest that the telltale signs of LLM usage can be identified by looking for these “marker words.” For example, a line like “A comprehensive grasp of the intricate interplay between […] and […] is pivotal for effective therapeutic strategies” highlights how LLM outputs often include such markers.
The researchers estimate that at least 10% of post-2022 papers in the PubMed corpus used some LLM assistance. This percentage could be higher, as their analysis might miss LLM-assisted texts lacking the identified markers. The study also found variation across different subsets of papers, with higher LLM marker word usage in papers from countries like China, South Korea, and Taiwan.
As Detectors Get Better, So Will LLMs
Detecting LLM use is crucial because “LLMs are infamous for making up references, providing inaccurate summaries, and making false claims that sound authoritative.” As awareness of LLM marker words grows, human editors may become more adept at removing these words from generated texts, making LLM detection harder. Future LLMs might even conduct frequency analysis to mask their outputs better, necessitating new methods to identify AI-generated text.
To avoid detection, removing or reducing the use of these excess words in your Gen AI writing is key.