AI application security

What is an AI application?

An AI application is a software system that uses an artificial intelligence or machine-learning model as a core component to automate complex tasks. These tasks might require language understanding, reasoning, problem-solving, or perception to automate an IT helpdesk, a financial assistant, or health insurance questions, for example.

AI models alone may not be directly beneficial to end-users for many tasks. But they can be used as a powerful engine to produce compelling product experiences. In such an AI-powered application, end-users interact with an interface that passes information to the model, often with supplementary sources of data such as health insurance information for the customer using the insurance app. Alternatively, the models can be linking the application to external tools to be automated, such as submitting an insurance claim.

With billions of parameters and massive datasets, sophisticated pre-trained large language models (LLMs) provide general-purpose functionality. Companies may customize open-source and closed-source LLMs to improve the performance of a particular task using one or more of these techniques:

• Fine-tuning: An additional training task in which a smaller, specific data set is used to specialize the model.
• Few-shot learning: An approach used to supplement any queries to the model with examples of how to appropriately respond to requests; this is often included in the system instructions that describe to the model how it should operate.
• Retrieval-augmented generation (RAG): A technique used to connect to additional data sources, such as text files, spreadsheets, and code, through a vector database. These data are fetched dynamically for each query, wherein specific documents that relate to the query are used to supplement the prompt so that the model has the correct context to answer a specific question.

Examples of AI applications

The versatility of AI is helping to transform every industry. Common examples include:

• Chatbots: Enterprise search, customer service, virtual assistants
• Decision optimization: Credit scoring, algorithmic trading, setting prices, route selection
• Complex analysis: Predict outcomes, classification
• Content creation: Writing tools, image and video creation

What is AI application security?

AI application security is the process of making AI-powered applications safe and secure by hardening the underlying components against vulnerabilities and actively mitigating attacks. It encompasses the entire AI development lifecycle spanning the supply chain, development, and production. The concept of 'shifting left' means AI teams are considering security from the earliest stages, which can help to identify and remediate vulnerabilities at the time of development. Companies must also continuously monitor for novel threats and attacks in production.

As adoption accelerates, so does the frequency and sophistication of attacks on AI systems. Protecting AI without stifling innovation is the concern of cybersecurity and AI leaders alike. This shared responsibility requires an organization-wide approach to protect against safety and security risks. Companies need a novel approach. Companies need AI application security.

What are the key differences between AI application security and traditional AppSec?

The difference between AI application security and AppSec – or standard application security – is the application engine; AI application security focuses on AI, while AppSec is designed for traditional software. As traditional software is typically deterministic, meaning that the same output is always generated from the same input, it is highly predictable and consistent. This translates to a relatively stable and well-understood set of vulnerabilities covered by traditional AppSec.

Conversely, AI comes in various model types which are often non-deterministic. This is particularly true for generative models, where the outputs are routinely different when given the same input. Threats to non-deterministic models require different mitigation techniques than deterministic software. Additionally, some AI applications continuously learn from human feedback or other data, which means new vulnerabilities and emergent behavior can appear after deployment, unlike traditional software that doesn't change unless you change it.

AI presents unique challenges that necessitate a new paradigm to mitigate security and safety risks.

AI application security vs. AppSec: Differences at a glance

What are AI application security solutions?

While traditional cybersecurity solutions won't work on AI Applications due to the differences between AI and traditional software, many of the techniques and best practices can be reappropriated at each stage of the AI development lifecycle.

Securing AI Application Development

Model validation consists of testing a model’s susceptibility to adversarial inputs like prompt injections and jailbreaks that can elicit unsafe or insecure outcomes. Whether using open-source, commercial models or proprietary models, it is necessary to validate the model for potential security and safety issues. For example, fine-tuning a well-aligned foundation model can destroy alignment in ways that must be understood by the application developer. The only requirement is API-or black box-access to a model. This process should be repeated every time there are changes to a model.

Secure the AI supply chain

The AI supply chain is every model, dataset, and software dependency your application uses. Like the traditional software supply chain, the AI supply chain includes software dependencies from the initial stages of development. But it also often involves third-party training data and model components, such as open-source models from repositories like Hugging Face. Every component of these applications may pose a unique risk for the system.

Scan these models for unsafe or malicious code in the file formats. Test each model for unsafe or insecure behaviors before using the application. Repeat these assessments every time the model changes.

The table below shows the components of AI applications, the risk they pose and how to take control.

Securing AI application deployment

Once an AI application is put into production, it’s important to continually scan the AI application for emerging safety and security vulnerabilities. These vulnerability scans should be informed by ongoing threat intelligence to ensure that the application is not susceptible to new attacker techniques.

Block attacks with an AI application firewall

An AI application firewall inspects traffic in both directions, serving as a runtime control. The firewall enables AI application developers to block malicious requests and undesired information from reaching the model, as well as unsafe responses from reaching the end-user. Unlike traditional software vulnerabilities, AI vulnerabilities cannot be patched, so they must be controlled. The firewall logs every blocked request and response; these logs can be passed to a security ticketing system and a security information and event management (SIEM) solution to be evaluated as part of the company's preferred security workflow.

How Cisco can help

Cisco AI Defense is an end-to-end solution trusted by enterprises worldwide to protect both the development and use of AI applications so they can advance their AI initiatives with confidence.

FAQ

What is prompt injection and how do you address it?

Prompt injection is an attack in which someone writes input that makes a model ignore its instructions and do something unintended. In a direct attack, the attacker types the malicious prompt. In an indirect attack, the instructions hide in content the model reads, such as a web page or document. You can't patch this weakness out of a model, so you control it. Validate the model against adversarial inputs before you deploy, and inspect prompts and responses at runtime with an AI application firewall. AI security and safety taxonomy defines both types and maps each to mitigations.

How do you secure AI agents?

AI agents answer questions, plan tasks and take actions through connected tools. This means that a successful attack can cause serious changes in your systems. Limit each agent to the tools and data it needs, and require human-in-the-loop approval for high-impact actions. Runtime guardrails inspect what the agent receives and what it tries to do, creating logs of every action. You can learn more about Cisco AI Defense, including details about how AI chatbots and AI agents work and where the business value is.

How do you secure a RAG application?

A retrieval-augmented generation (RAG) application pulls documents from your data sources into the prompt for each query, so those sources become vulnerable to attacks. Poisoned documents can carry hidden instructions, and weak access controls can expose sensitive data in a response. Vet and monitor what enters the knowledge base, enforce access controls on the source data, and inspect responses before they reach users. You can explore more insights on RAG applications, including how to secure these applications.