Generative artificial intelligence can create text, images, music, code and speech, by drawing on existing content and data. Current examples include ChatGPT, DALL-E and Jukebox. The content is delivered to the user through interaction with a chatbot and question-and-answer interface.

While nascent, the technology is being tested widely for applicability in business. However, generative AI applications entail considerable deployment risks, necessitating strong risk-mitigation techniques. 

There’s been a tremendous surge of interest in generative AI for business use since ChatGPT was introduced in November, 2022. A plethora of startups are offering services, and venture funding is pouring in. A key question at senior leadership and board levels today is “What is your AI strategy?” Organizations large and small have either tiptoed or dove headlong into generative AI. Lawsuits over the protection of information have already begun, including a class action against Google over alleged misuse of personal information during training of the company’s Bard chatbot.  

Business functions that are exploring generative AI include sales and marketing, supply chain visibility and predictability, customer service and account management, streamlining of tasks and proactive identification of bottlenecks, logistics and distribution, IT and legal.

How Generative AI Works

Neural network-based large language models (LLMs) form the basis of generative AI. A deep-learning neural network consists of multiple layers of interconnected neural nodes. ChatGPT receives a text string as input, encodes it into numerical data, feeds it into the network, then generates a response. The output takes the form of one word at a time, with each new word dependent on the previous one. This is based on a probability calculated by the model; that is, the word with highest contextual probability is the ultimate output.

The model is initially trained with supervised learning to recognize data patterns. In the next stages, it’s refined through reward and reinforcement. ChatGPT’s training dataset essentially is the entire available public data, consisting of upwards of 100 trillion data parameters for GPT-4.

Google, Microsoft, Open AI, Meta, DeepMind and Nvidia are some of the leading LLM developers. These models, while in development and test mode, are being made available for a fee to users via an application programming interface (API).

The rise in use of LLMs in daily life and business has exposed the deficiencies of their outputs, as well as risks of data leakage and poisoning, the stealing of intellectual property, and system hijacks. In addition, chatbots are subject to “hallucinations,” whereby they make up answers and present false information. The willingness of users to trust the tool without knowing its underlying operations and risks amplifies this problem.

How different is the risk of using an LLM compared with other cloud-based applications such as enterprise resource planning (ERP)? With traditional applications, there’s no need to share organizational data for model refinement and training. ERP systems are confined and don’t entail the risk of training-related data leakage. All transactions and sensitive business data are exchanged using current security protocols. Cloud-based applications also don’t encounter the risk of repeated training due to the ever-increasing use of data sources with no sensitivity guardrails. Generative AI’s dependence on an ever-increasing data corpus, by contrast, increases the possibility of LLM memorization data leakage and external hacks. These risks are further exacerbated when cross-functional processes are integrated and generative AI is used. It’s possible that sensitive data can be leaked during interactive Q&A sessions. 

Internal and External Risks

Fine-tuning the model with sensitive organization data is risky, as is direct employee input of sensitive data into LLM training models. Samsung reported employees inputting confidential information, including highly sensitive source code, into an AI chatbot while attempting to correct coding errors. With no user understanding of the internal operations of an LLM, it’s unknown how fine-tuning data is used, stored, and outputted by the model. The act of fine-tuning is devoid of traceability of the answer; it becomes impossible to control and limit documents and information to certain users or groups, and costs are high.

External risks include data poisoning through the corruption of training data by attackers. Prompt inputs can be manipulated to generate malicious outputs and gain illegal access to data. Where multiple open-source data and AI applications are involved, the potential for damage from hackers is enormous.

Following are some major risks from training and data use in the application of generative AI.

Training. Data leakage can occur when the model predicts a full set of data upon being given a subset. If, for example, there’s a query to retrieve the last four digits of a credit card, the returned information may include the entire card number. In the case of an LLM, even if sensitive data isn’t part of the training input, there could be contextual data that is extracted and outputted.

Jailbreaks. The model might be asked to respond as a bad actor, and behave in that manner. It can be instructed to role-play the exact opposite of the conventional output the generative AI is trained for.

Data extraction. This occurs when back-end query attacks generated by ChatGPT are injected to extract data or generate harmful output.

API risks. Trojans and prompt injections can be used to trigger nefarious activities, such as an impersonator reading or composing e-mails, or sending messages via users’ address books. Such injections can also allow infiltration into other API-integrated LLMs. At this scale, this can almost impossible to know the source of the compromise. 

Source code hacks. Malicious, obfuscated code is inserted in a code block, which the developer may or may not check. Execution of the code results in the propagation of breaches and data exfiltration.

What to Do About It

Following are some ways to mitigate these risks.

Understand each type of risk, and its potential for breach.

Identify where to use generative AI. Stakeholders need to think carefully about the impact and cost of false positives and negatives.

Ensure visibility, led by the chief information security officer and IT, of planned and deployed projects or applications within the organization. Monitor each deployment for its success rate, along with its accompanying data and process risks.

Create organizational guardrails and policies for deploying generative AI applications. Decide which infrastructure stack to use — for example, private versus public cloud.

Clearly identify and understand cross-functional data sharing. Provide controls or blocking for sensitive or confidential data. Validate organizational data for model fine-tuning against the inadvertent release of sensitive data, and mask or anonymize all such data.

Conduct security checks and tests before deploying a fine-tuned model. This includes AI-based tools for detecting system anomalies and bad actor behaviors. Also include capabilities to test model output — for example, does a medical summary contain other patient information or deductions?

Develop processes and tools to validate the legitimacy of generative AI-supplied codes.

Use private ChatGPT. It’s possible to train a private version with an organization’s own data and inputs. Either a pre-trained model or a ground-up built model and fine-tuning can be used with corporate data. Separate the LLM from the corporate knowledge base. However, creating a private ChatGPT requires technical expertise and large datasets and resources, and might be unaffordable.

With generative AI and LLMs in their infancy, guardrails and protective policies are still evolving. Until effective AI-based risk-mitigation tools and policies are in place, organizations should take a careful approach to deployment, and be mindful of the enormous risks at stake.

Shubho Chatterjee is a partner at Empirical Consulting Solutions.