Sounding sure isn't being right

What it is

When an AI chat tool states something false as if it were true, people call it a hallucination. It isn't lying on purpose; it's predicting words that sound right. The tone is the same polished confidence either way.

Why it matters

Made-up facts, quotes and references can look completely real, and people have been caught out. Separately, what you type may be kept and used to improve future versions, so it's worth deciding what you share before you share it.

Key points

• A hallucination can be a fact, a number, a quote or a reference. References are a common one: they can have real-looking authors and titles and still not exist.
• In a 2023 study, more than half of the references one AI chat tool wrote were made up, and nearly 1 in 5 from a newer version. Those are dated numbers, not today's rates, but the habit of checking still matters.
• In 2023 a US court sanctioned lawyers who filed fake case citations that an AI chat tool had made up.
• Three quick checks: Where did this come from? Can I confirm it somewhere else? Would someone who knows this field agree?
• Treat an AI chat like a friendly stranger on the bus: chat away, but don't hand over your wallet. Leave out passwords, account numbers, health records and other people's private details.
• Your chats may help train future versions, on free and paid plans alike. Many tools let you choose. Paying doesn't automatically opt you out, and free doesn't mean you have no say. Check your settings rather than assuming.
• Build trust on low-stakes things first, like a summary of an article you've already read, so you can see where it slips.

Words to know

AI chat tool

A program you type or talk to that writes back in everyday language.

hallucination

When AI states something false as if it were true.

citation

A note pointing to where a fact came from, like a book or court case.

train

To teach an AI by feeding it lots of examples, sometimes including people's chats.

Sources

• In a 2023 study, 55% of the citations from one AI chat tool version and 18% from a newer version were fabricated. Fabrication and errors in the bibliographic citations generated by ChatGPT (Walters & Wilder, Scientific Reports, 2023) (opens in a new tab)
• Attorneys were sanctioned in 2023 for using fake case-law citations generated by an AI chat tool. Mata v. Avianca, Inc. - Wikipedia (opens in a new tab)
• A consumer AI plan's training choice applies to free and paid plans alike, and can be changed at any time. Updates to Consumer Terms and Privacy Policy (Anthropic) (opens in a new tab)
• Another major AI assistant offers an activity setting that controls whether chats are kept and may be reviewed. Gemini Apps Privacy Hub - Gemini Apps Help (opens in a new tab)