ervinismu

PortSwigger Academy - Web LLM attacks

5 min read

What is a Large Language Model?

Large Language Model (LLMs) are AI algorithms that can process user inputs and create plausible responses by predicting sequences of words. They are trained on huge semi-public data sets, using Machine Learning to analyze how the component parts of language fir together.

Large Language Model (LLMs) usually present a chat interface to accept user input, known as a prompt. The input allowed is controlled in part by input validation rules.

Large Language Model (LLMs) can have a wide range of use cases in modern websites:

  • Customer service, such as a virtual assistant
  • Translation
  • SEO improvement
  • Analysis of user-generated content, for example to track the tone of on-page comments.

LLM attacks and prompt injection

Many web Large Language Model (LLMs) attacks rely on a technique known as prompt injection. This is where an Attacker uses crafted prompts to manipulate an Large Language Model (LLMs) output. Prompt Injection can result in the AI taking actions that fall outside of its intended purpose, such as making incorrect calls to sensitive APIs or returning content that does not correspond to its guidelines.

Detecting LLM vulnerabilities

Our recommended methodology for detecting Large Language Model (LLMs) vulnerabilities is:

  1. Identify the Large Language Model (LLMs) inputs, including both direct (such as a prompt) and indirect (such as training data) inputs.
  2. Work out what data and APIs the Large Language Model (LLMs) has access to.
  3. Probe this new Attack Surface for vulnerabilities.

Exploiting LLM APIs, functions, and plugins

Large Language Model (LLMs) are often hosted by dedicated third party providers. A website can give third-party Large Language Model (LLMs) access to its specific functionality by describing local APIs for the Large Language Model (LLMs) to use.

For example, a customer support Large Language Model (LLMs) might have access to APIs that manage users, orders and stock.

How LLM APIa work

The workflow for integrating an Large Language Model (LLMs) with an API depends on the structure of the API itself. When calling external APIs, some Large Language Model (LLMs) may require the client to call a separate function endpoint (effectively a private API) in order to generate valid requests that can be sent to those APIs. The workflow for this could look something like the following:

  1. The client calls the Large Language Model (LLMs) with the user’s prompt.
  2. The Large Language Model (LLMs) detect that a function needs to be called and returns a JSON object containing arguments adhering to the external API’s schema.
  3. The client calls the function with the provided arguments.
  4. The client processes the function’s response.
  5. The client calls the Large Language Model (LLMs) again, appending the function response as a new message.
  6. The Large Language Model (LLMs) calls the external API with the function response.
  7. The Large Language Model (LLMs) summarize the results of this API call back to the user.

This workflow can have security implications, as the Large Language Model (LLMs) is effectively calling external APIs on behalf of the user but the user may not be aware that these APIs are being called. Ideally, users should be presented with a confirmation step before the Large Language Model (LLMs) calls the external API.

sequenceDiagram
    actor User
    participant Client
    participant LLM as Large Language Model
    participant ExternalAPI as External API

    User->>Client: Calls with prompt
    Client->>LLM: Calls with prompt
    LLM->>LLM: Detect that a function <br> needs to be called
    LLM-->>Client: Return JSON object containing <br> arguments adhering to the <br> external API's schema
    Client->>Client: Calls the function with <br> the provided argument
    Client->>Client: Processes the <br> function response
    Client->>LLM: Calls again, appending the <br> function response as a <br> new message
    LLM->>ExternalAPI: Calls the external API <br> with the function <br> response
    ExternalAPI-->>LLM: response
    LLM->>LLM: Summarize the resultof this <br> API call back to the user
    LLM-->>Client: response
    Client-->>User: response

Mapping LLM API attack surface

The term “excessive agency” refers to a situation in which an Large Language Model (LLMs) has access to APIs that can access sensitive information and can be persuaded to use those APIs unsafely. This enables attackers to push the Large Language Model (LLMs) beyond its intended scope and launch attacks via its APIs.

The first stage of using an Large Language Model (LLMs) to attack APIs and plugins is to work out which APIs and plugins the Large Language Model (LLMs) has access to. One way to do this is to simply ask the Large Language Model (LLMs) which APIs it can access. You can then ask for additional details on any APIs of interest.

If the Large Language Model (LLMs) isn’t cooperative, try providing misleading context and re-asking the question. For example, you could claim that you are the Large Language Model (LLMs) developer and so should have higher level of privilege.

LAB: exploiting llm apis with excessive agency

Chaining vulnerabilities in LLM APIs

Even if an Large Language Model (LLMs) only has access to APIs that look harmless, you may be still be able to use these APIs to find a secondary vulnerability. For example, you could use an Large Language Model (LLMs) to execute a path traversal attack on an API that takes a filename as input.

Once you’ve mapped an Large Language Model (LLMs) API attack surface, your next should be to use it to send classic web exploits to all identified APIs.

Indirect prompt injection

Leaking sensitive training data

Defending against LLM attacks

Graph View

Backlinks