Knowing when not to trust it

That rule covers most of the risk in this kind of work. The rest of this page is the detail.

Goes wrong predictably

Citations and references. The most dangerous one, because the output looks so right. Plausible authors, real journals, correct formatting, and the article doesn’t exist. Or it exists and says something else. Never let a tool supply a reference for anything you’ll put in front of learners. Find the reference yourself and paste it in.

Numbers inside sentences. Percentages, prevalence, denominators, “approximately”. Even when it’s working with your own data, figures get swapped, rounded, or attached to the wrong label. Every digit in a finished document needs to trace back to something you supplied.

Counting long lists. “How many of these 300 comments mention scheduling?” gets you a confident approximation. If the count is going into a report, split the list and count it yourself, or check with a search.

Anything about a regulation, standard or accreditor. It has a general sense of what accreditation frameworks look like and no reliable grasp of what your accreditor currently requires. It’ll paraphrase criteria fluently and get the substance slightly wrong. Check requirements against the source, every time.

Anything recent. Guidelines revised last quarter. A standard updated this year. A tool’s own current features. Treat recent as a warning sign by default.

Arithmetic across a document. Session timings that don’t add up to the session length. Percentages over 100. An attendance figure that contradicts the completion figure three paragraphs later. Add things up yourself.

Reliable enough to lean on

Reorganizing material you supplied. Reordering, reformatting, notes into prose, prose into bullets, a spreadsheet into a paragraph.

Tone. Making something more formal, warmer, shorter, less promotional. Matching a writing sample.

Giving you options. Five draft objectives, three openings, two versions of an email. You’re the one choosing, and choosing is safe.

Spotting what’s missing or repeated. “Which parts of this gap statement aren’t covered?” “Which of these are the same point twice?” Genuinely good at this, and it’s hard to do on your own work.

First drafts of anything you’d otherwise sit and stare at. The blank page problem is real, and this dissolves it.

Being a difficult conversation partner. See recipe 06.

Confidence tells you nothing

This is the most important thing to get calibrated: how sure it sounds has no relationship to whether it’s right.

The invented citation is formatted more beautifully than the real one. The wrong count is stated with no more hesitation than the right one. There’s no tone of voice to listen for and no hedging that reliably signals uncertainty.

Everyone learns this the hard way once. It’s cheaper to learn it here.

Three habits worth making automatic

The number sweep. Before any document leaves your hands, look at every figure and point at where it came from. This routine catches common high-consequence errors in reports.

The quote search. Any quote in an output gets searched for in the source file. Paraphrase presented as a quotation is common and easy to catch.

The source check. Any claim about your accreditor’s requirements gets checked against your accreditor, not against the tool.

Where the line sits

A useful way to hold it: these tools are excellent at shape and unreliable at fact.

Give them the facts and ask for the shape, and they’re one of the more useful things to land on a coordinator’s desk in twenty years. Ask them for the facts and you’ve built yourself a very persuasive liability.

Every recipe on this site is that one sentence, applied.

Try this

Ask a tool for three references on a topic you know well. Then go and look for them.

Do it once. Finding out which ones don’t exist is worth more than reading about it, and you’ll never quietly trust a supplied citation again.