Securing GenAI: Challenges

The New Electricity[1]

AI has been called the New Electricity that is changing the way we do everything. Large Language Models (LLMs) and Generative AI (GenAI) solutions are increasingly being adopted in businesses, governments and other organizations. With the GenAI helping automate a large segment of work it is not surprising that 90% of the Fortune 500 companies are already using OpenAI’s technology[2]. Well over half the leaders (64%) in companies with 1000 employees or more, feel a sense of high urgency to adopt generative AI, but they are also facing a shortage of highly skilled workforce required to go ahead with the adoption[3]. It has been only a couple of years since the GenAI fever has spread through the industry. It is estimated that about 2 million software developers from Fortune 500 companies are developing on OpenAI’s APIs alone[2]. OpenAI is the most popular, but adding up for all the other LLMs out there, the number would be easily more than double of that. That is a very large workforce developing very powerful tools which handle data of all sorts that touch the lives and everyday working of people and organizations. This would include our personal details, financial and health information, education information and much more.

FOMO & The Lack of Caution

Fueled by the fear of missing out (FOMO) on the hype created by this new technology, GenAI is being introduced rapidly into not just the business processes, but also health and education. In this race to deliver the functionality fast, security has become an afterthought. We have seen examples of some systems where secure design and implementation was traded off for speed.

Thermal Exhaust Ports [4]

While GenAI tools are revolutionary in their capabilities, they also bring out new vulnerabilities and ways for threat actors to exploit the systems.

These vulnerabilities could have severe consequences. For example, consider a scenario where a logistics company uses an LLM-powered system for route optimization and inventory management. A threat actor using a prompt injection attack could manipulate the routing system to reroute high value cargo to unauthorized locations. They could disrupt the supply chains with false information about inventory levels and cause significant financial losses.

A potential exploit could involve sensitive information disclosure. In absence of proper security measures, a cleverly designed prompt on an AI powered customer service system could lead to revealing shipping details or customer information. That would be a violation of customer privacy, break regulations and worst of all erode the customer’s trust.

OWASP top 10 for LLM

The Open Worldwide Application Security Project (OWASP) has been publishing and updating the most common risk for Web Applications, IoT and System Software since 2003. This non-profit has now published its Top 10 vulnerabilities for the applications using LLMs [5]. It lists the top 10 vulnerabilities in detail and also prevention and mitigation strategies. Here's a brief overview:

1. Prompt Injection: Manipulating input to make the model behave unexpectedly.

2. Insecure Output Handling: Failing to properly sanitize or validate model outputs.

3. Training Data Poisoning: Compromising the model's training data to introduce biases or vulnerabilities.

4. Model Denial of Service: Overwhelming the model with requests to disrupt service.

5. Supply Chain Vulnerabilities: Risks associated with pre-trained models or third-party components.

6. Sensitive Information Disclosure: Unintended revelation of confidential data.

7. Insecure Plugin Design: Vulnerabilities in extensions or plugins for LLMs.

8. Excessive Agency: Giving AI models too much autonomy in critical decisions.

9. Over-reliance: Excessive trust in AI outputs without human oversight.

10. Model Theft: Unauthorized access or reproduction of proprietary models.

When you have a hammer ...

Another dimension in securing your organization’s data is protection from the employees leaking data unintentionally while using the publicly available LLMs to speed up their work. With the hammer in hand, everything looks like a nail and we end up making a lot of holes.

There have been instances of engineers leaking proprietary code by uploading it to ChatGPT [6]. As a result Samsung banned the use of ChatGPT and other chatbots company wide. JPMorgan Chase also imposed restrictions on the use of LLMs.[7] This was followed by similar restrictions by several major financial institutions - Bank of America, Citigroup, Deutsche Bank, Wells Fargo and Goldman Sachs [8]

With great power comes great responsibilities

We are building extremely powerful systems with AI that have far reaching impacts on every aspect of our lives, which will continue to get deeper and wider with time. When exploited by threat actors, the blast area will be of comparable size both deep and wide. So it becomes our responsibility to make ourselves aware of the risks, keep up to date with the threats and do everything possible to secure the systems we build.

In the next article in this series we will talk about assessing and mitigating the risks in GenAI systems.

References:

[1] SCET University of Berkeley, Oct 2023 “AI is the New Electricity”: Insights from Dr. Andrew Ng

[2] Financial Times, Feb 2024 OpenAI on track to hit $2bn revenue milestone as growth rockets

[3] Google Cloud Blog, May 2023 The Prompt: We asked business leaders what they’re expecting from generative AI, (retrieved 2024-07-09)

[4] Wookiepedia, Thermal Exhaust Port

[5] Open Worldwide Application Security Project (OWASP), OWASP Top 10 for LLM Applications

[6] Bloomberg, May 2023 Samsung Bans Staff's AI Use After Spotting ChatGPT Data Leak

[7] Bloomberg Feb 22, 2023, JPMorgan Chase Restricts Staffers’ Use Of ChatGPT

[8]Bloomberg, Feb 24, 2023 , Wall Street Banks Are Cracking Down on AI-Powered ChatGPT