AI Hallucinations Test the Old Adage That the Customer Is Always Right

Madison, a server in New York City who withheld her last name to protect her identity, told The Verge she asks every table about allergies. She said close calls have become more common. Sometimes a customer says they have a shellfish allergy, then orders a fish dish served with a broth made from shellfish. When she warns that a dish contains an allergen, Madison said, guests have pushed back and said ChatGPT disagrees. She recalled telling customers, in effect, that they are allergic to shellfish, ChatGPT says there is no shellfish, and the restaurant says there is.

At the Michelin-recommended Italian restaurant where she worked until recently, Madison also saw increased reliance on generative AI among patrons. She said she often suggested bringing the sommelier, only to be told the guest would ask ChatGPT instead. Sommeliers have been dismissed from tables because guests preferred ChatGPT, she said. Customers have requested bottles the restaurant does not carry or, in some cases, bottles that do not exist. Madison said the hallucinations worsen preexisting anxieties about wine. Some guests already worried that sommeliers were like used car salesmen or would think they were stupid, she said. To soften the blow of AI hallucinations, Madison and colleagues would say the wine list is hyper-specific and maybe the app is showing a French wine when the restaurant does not carry French wine, rather than telling the customer the chatbot invented a brand.

The Verge reported that AI creep now appears ubiquitous in customer service jobs. The launch of Muse, Meta’s new AI agent, will likely exacerbate the issue and add headaches for workers fielding nonsensical requests, the report said. Movie theater workers, baristas and Apple Store employees all described customers asking for products or features that do not exist. On Reddit and X, customer service workers complained about customers who take ChatGPT’s word as gospel.

Jason, a river ranger who works on federal lands in the Mountain West and asked to be referred to by a pseudonym because he is a federal employee, said he has noticed more visitors arriving with nonsensical AI-generated itineraries. He and colleagues have explained that the AI hallucinated campsites or scheduled things out of geographical order, so groups would have to row their boats upstream some days, which is not possible, and float with the river current on other days. Jason said that even when a ranger explains this while pointing at a map, multiple groups have remained skeptical. He said a certain type of AI user is inclined to trust ChatGPT over rangers, maps and printed guidebooks.

Casey, an after-school program teacher in New York who asked to be referred to by a pseudonym, said many students’ parents, nannies and other caretakers now turn to ChatGPT or Claude for questions about childhood development. One family consulted ChatGPT for a bedtime routine and sleep schedule, but their child was not getting enough sleep, Casey said. Based on the child’s behavior in the program, Casey estimated the child needed two to three more hours of sleep a night and told the parents. The parents insisted that their ChatGPT told them the schedule was fine. Casey said some parents now see generative AI tools as experts whose knowledge surpasses that of childcare professionals. It can be hard to talk with someone who has made up their mind because they trust this outside entity as invaluable, she said. Casey also noticed more ordinary uses, such as kids coming in with AI-generated coloring sheets.

Lauren Wilford, who has worked in food service for 15 years, has noticed AI creeping into recent guest interactions. The Verge reported that it started with the menu: someone would upload a photo of it to ChatGPT or Claude and ask what to order, or for guidance on a wine-related question. The report described a broader pattern in which customers use AI-generated answers to challenge workers who provide direct, localized information.