How do you spot text generated by LLM/AI

Since we are testers we tend to spot pattern and issues even when we do not want to :smiley: .

So i was wondering how do you spot a text that is generated by an LLM/AI.
Please do not say I use a scanner.

For me the tells are usually these 3

  • Using the prashe “In the world of”
  • Using words that one does not usually use in a day to day conversation like delve
  • The post is marked with emojis all the way.

What about you?

2 Likes

@restertest AI generally uses too many adverbs and adjectives to make its point.

1 Like

I actually do these 2 myself xD
Damn I might be a robot! :robot:

2 Likes

I knew it @kristof .

That explains it how you are so good at security.

You talk to the computer in robot languge.

Got you :smiley:

2 Likes

It appears very polished

what I have noticed is that in the ChatGPT response, the semicolon is inserted between lines, so wherever I noticed a semicolon between lengthy lines, I understood that it is copy pasted from ChatGPT, because usually, we use a comma or full stop in real.

Very complex words are visible which we usually don’t use in day-to-day life.

Instead of “are”, the apostrophe re is visible. like we’re, you’re,

3 Likes

Two main things stand out, I don’t know 100% how to describe it, but it’s super obvious when I see it. The paragraphs are broken up almost like fleshed out dot points? Like each one was a brief thought that has been expanded upon. And the last one is always a clear conclusion paragraph - much of the time even starting with the phrase “In conclusion, xyz”. It reminds me of high school essay writing, very formulaic.

2 Likes

I think it’s because people write like they talk and like LLM systems write to polished and doesn’t feel like somebody talking.

1 Like

Why?
Some posts/texts/messages have this artificial vibe and sometimes I don’t even think about why exactly and I don’t really care if they are AI-generated or not.
Another point is that many of the characteristics mentioned above are just features of people for whom English isn’t their native language but they have quite a good level of studying it using books and other learning materials, passing exams rather than using it naturally in conversations with native speakers. People from some regions have quite a sophisticated language but again they don’t use it naturally so sometimes I can say that it sounds like a person from a particular country (won’t mention the country because it might be stereotypic).
Additionally, using some tools to correct mistakes and improve readability like Grammarly may lead to adding this AI scent to the text. Is it okay for you?

I’m writing this because there might be lots of different cases (AI might be used in various ways even to develop unique ideas in certain ways which won’t make the text completely AI-generated but it may look like so) and what it may seem at the first glance doesn’t mean a lot, even if it’s completely AI-generated. Better concentrate on the content, ideas, and usefulness, and if you need to determine for sure if a text was AI generated (e.g. maybe you’re a teacher) then use particular tools and methods not just your intuition.

PS: Was this text AI-generated? Or AI-enhanced? Does it sound natural to you?

As one of those rare people who tries to keep the semicolon alive, I think you might be right.