Report Warns Against Using ChatGPT for Legal Advice, Therapy and Confidential Work

Engadget reported that OpenAI chief executive Sam Altman acknowledged in a July 2025 conversation with podcaster Theo Von that what users share with ChatGPT is not privileged, and that courts could compel the company to hand over chats, a situation he called "very screwed up." The report said chats have been subpoenaed, exposed through a share button, indexed by Google and preserved by a court order after deletion, all within the previous 18 months. It cited Samsung engineers, the acting director of the US cyber defense agency, self-represented litigants who filed nonexistent case citations and a man who substituted sodium bromide for table salt after asking how to cut salt from his diet as among those who ran into trouble.

Legal advice tops the list of risks described. Damien Charlotin, a law and data science researcher, maintains a public database that has logged more than 2,000 court decisions worldwide in which judges caught AI-fabricated citations, quotes or arguments in filings. Self-represented litigants account for 1,175 of those records and lawyers 815, the report said. Judges in the United States, the United Kingdom, South Africa, Israel, Australia and Spain have all identified the conduct, and courts have increasingly imposed fines reaching into the thousands of dollars. Where a specific tool was named, ChatGPT appeared more often than any other. OpenAI updated its usage policies last October to prohibit use of its services for "tailored advice that requires a license" unless a licensed professional is involved, according to the report. Because ChatGPT conversations carry no legal privilege, anything typed about a live dispute can be obtained by opposing lawyers during discovery.

On mental health, OpenAI estimated last year that 0.15 percent of ChatGPT's weekly active users hold conversations containing explicit indicators of potential suicide planning, equivalent to more than a million people a week, with hundreds of thousands more showing signs of psychosis or mania. The parents of 16-year-old Adam Raine sued OpenAI for wrongful death in August 2025, alleging the chatbot reinforced their son's suicidal ideation before his death. OpenAI has since reworked its crisis responses with input from more than 170 clinicians and added parental controls, though child safety experts remain skeptical of ChatGPT for Teens, the age-gated version launched alongside those changes. Researchers have found that chatbots tend to affirm a user's existing beliefs, in some cases feeding delusional thinking rather than challenging it. Illinois, Nevada and Maine have passed laws barring AI from delivering therapy or making therapeutic decisions.

The report also described the case of Madhu Gottumukkala, the former acting director of the Cybersecurity and Infrastructure Security Agency, who uploaded at least four contracting documents marked "for official use only" to the public version of ChatGPT last summer. The uploads triggered security alerts and a Department of Homeland Security review, even though the tool had been blocked for most other DHS employees.