The problem with so much generative artificial intelligence today goes beyond just giving wrong answers. It’s when the errant AI model is certain that it’s right.

Up to now, ignorant overconfidence has been the exclusive provenance of humans. But AI is now mimicking that trait. It can deliver the most ludicrous statements — glue on pizza, anyone? — with a sense of total authority. That’s a side effect of the way in which AI models have been programmed to provide “decisive and clear answers,” says Niranjan Krishnan, head of AI solutions for FPT Americas, a provider of business software and consulting services.

It’s no joke when AI gets it wrong. Krishnan views the problem of machine overconfidence as “one of the most dangerous and overlooked risks in enterprise AI.” What’s more, it becomes increasingly serious as companies scale their dependence on autonomous AI agents. Often their errors are tough to detect — until they end up exposing companies to punitive regulatory action or eroding customer confidence.

Krishan says the latest iteration of AI “is far more articulate than most human beings,” combining a facility with language and deep pool of knowledge trawled from the internet to deliver polished answers — sometimes backed by sources that are entirely made up.

Take companies’ increasing reliance on AI to help customers troubleshoot product issues. The agent can be overeager to help, ignoring subtle differences between product types and delivering faulty diagnoses. It might even offer guidance in areas for which it was never trained, Krishnan says.

An obvious solution to the problem is to train the model to confess when it doesn’t have a suitable answer. But programming it to do that isn’t so simple — as with humans, it might not know what it doesn’t know. A better way is to ground the program in statistical modeling, declaring how confident the AI is in its conclusions. (A similar approach is often taken by human consultants, when they seek to extend current trends into future scenarios, accompanied by stated confidence levels.)

Fine — so an AI model says it’s “90% confident” in an answer. But humans are still needed to render a final judgment on the output. Krishnan says it’s important to employ certain deduction techniques to smoke out hallucinations. Reference-checking can verify the output by comparing it against third-party data sources that the model hasn’t seen. Independent verification also can determine whether the AI’s response is grounded in the actual input data.

But even the best AI detection programs aren’t foolproof. “They can only minimize or reduce, but not eliminate, the risk,” says Krishnan.

The same conundrum exists when it comes to recognizing AI-created deepfakes, which are growing ever more sophisticated as detection programs scramble to keep up. Also at issue is the growing problem of undisclosed AI-generated text. Certain “tells” exist, in the form of overreliance on bullet points and constant use of the “It’s not X, It’s Y” construction. But that method, too, is imperfect. Was this article written by AI? (It wasn’t – but isn’t that exactly what an overconfident AI bot would say?)

One trick is to ask the model the same question multiple times, then see if the answer comes out the same. “If it’s hallucinating, the responses will be very different from each other,” Krishnan says, adding that AI doesn’t actually have a bias toward being wrong. It could also be valuable to employ multiple large language models to address the same task, to see if all agree.

The problem worsens with scale, leading to the possibility of AI “model collapse” as answers become increasingly generic and irrelevant to the original prompt. Depending on how the model is structured, AI-generated data for building, reasoning and taking action can grow “worse and worse” over time.

Krishnan stresses the need for building both input and output guardrails into the core model, before any enquiries are made. AI responses need to be anchored to a company’s internal knowledge, product documents, user manuals and other traditional sources of information.

AI models may grow less flawed as they develop in the years ahead, but structural guardrails and human oversight must always be in place, Krishnan says. “As a practitioner, we are cautiously optimistic that we do everything to make sure the AI does what it should. Buying a car, you want to look at the engine. But you want to look at the brakes, too.”