AI Hallucinations Strike When Lawyers Need Help Most

A new paper by scientists and engineers reveals that AI hallucinations are not random glitches but foreseeable engineering failures. GenAI systems are most likely to fabricate information precisely when handling complex, novel legal questions with sparse precedent — the exact moments lawyers need reliable answers most. The danger is compounded by a false sense of security. AI tools perform accurately on straightforward, well-established information, building user trust before failing on harder questions. Lawyers who verify early outputs and assume later ones are equally reliable may unknowingly submit briefs filled with fabricated citations and flawed legal arguments.