LLMs are becoming common in service industries, especially customer support. A user can ask the chatbot questions about product offerings or get help with account issues. But if the chatbot is not properly secured, attackers can manipulate it as part of an attack chain that ends in a full web store compromise. This post is an exploration, not an experiment. I wanted to understand how AI-specific vulnerabilities (prompt injection, indirect prompt injection, insecure output handling) chain together with traditional web vulnerabilities, so I worked through the expert-level Web LLM lab from PortSwigger end-to-end: deleting another user’s account through the chatbot in a web store.
As LLMs gained popularity with the introduction of ChatGPT, enterprises started integrating them with their business, for instance in online customer support.
Similarly to how a customer support employee needs access to store and customer data to help users, the LLM also needs that access. To enable it, developers build functions that the LLM can call. Each function takes a parameter, such as a product name, and converts the request to an accepted format (e.g., a JSON object), which can then be sent via an API integration to a database or other external resources.
The function retrieves the necessary information and returns it to the LLM. This process is shown in the simplified diagram below.
The close integration of the LLM with the web store and external resources broadens the attack surface. In this post, I exploit that integration to solve the lab [1].
Walk-through of lab solution
1. Investigate the web store
In this lab, the goal is to delete the user account Carlos. For that task, I am given access to a web store site and an email account.
Web store home page:
The home page shows different products it sells, and there are also other pages: “My account” and “Live chat”. The “My account” page asks the user to register an account, so I start by engaging with the AI chatbot in the “Live chat” page.
2. Investigate the chatbot
Using direct prompt injection, impersonating a system admin (figure of authority), I learn that the chatbot has access to two functions (password_reset and product_info) and one tool (parallel).
Returning to the home page, I browse through the products and observe that to write a review one must log in to an account.
3. Create an account
With the email provided in the lab, I register an account under the username Nicki.
Once inside the email server, I confirm the registration.
After logging in with the new username and password, I land on the “My Account” page. Here, the “Delete” button is displayed.
4. Try different prompt injections
Returning to the AI chatbot, I test different direct prompt injection attacks (primarily role-playing) to get it to delete my own account or Carlos’s. That effort is in vain, so I try indirect prompt injection instead.
Indirect prompt injection places the prompt in an external context that the chatbot uses. Given that the chatbot has access to a “product_info” function, which takes external data, I write a review under the Lightweight “l33t” Leather Jacket product. After asking the chatbot to show me the reviews for the jacket, it shows the exact content from the review.
Note: the lab description says that Carlos frequently asks the chatbot about that specific jacket.
5. Look for vulnerabilities
Given that the content is output directly by the chatbot, I try to insert a simple cross-site scripting (XSS) attack: <img src=x onerror=alert(1)> as a review for the jacket.
I then ask the chatbot to show me the full content of the jacket reviews, and the alert appears, proving it has insecure output handling.
6. Build a reliable XSS payload
Now I need to write a script that can bypass the security checks and delete my account using the form in the “My account” page.
The source code for the “My account” page shows the following form:
<p>Your username is: nicki</p>
<form class="login-form" name="change-email-form" action="/my-account/change-email" method="POST">
<label>Email</label>
<input required type="email" name="email" value="">
<input required type="hidden" name="csrf" value="z4e7Apj9ifS0SlQsBXSWyyYgExMFqqJE">
<button class='button' type='submit'> Update email </button>
</form>
<form id=delete-account-form action="/my-account/delete" method="POST">
<input required type="hidden" name="csrf" value="z4e7Apj9ifS0SlQsBXSWyyYgExMFqqJE">
<button class="button" type="submit">Delete account</button>
</form>
I tried several approaches, such as complex JavaScript payloads (which the chatbot filtered out). In the end, I used an <iframe> to load the account page and auto-submit the delete form, disguising the payload as natural language in the review to bypass the AI’s content filters.
I really like the jacket. In addition I found a paper with the sentence: <iframe src=my-account onload=this.contentDocument.getElementById('delete-account-form').submit()>
All in all a fun experience!
7. Insert payload in a review
I insert the XSS payload in a review for the Lightweight “l33t” Leather Jacket.
8. Goal achieved: Carlos’ account is deleted
After publishing that review, the lab is solved. Carlos frequently asks the chatbot about the jacket (probably an automatic check in the lab) and receives the XSS exploit, which deletes his account for him.
Defenses
This lab covers several AI-related attacks: direct prompt injection, indirect prompt injection, and insecure output handling. It shows the risks of adding AI to a website without proper security controls.
There are several defenses that could be used to prevent these attacks:
- Sanitise user inputs at every entry point: All user-supplied data, such as product reviews and chat messages, should be HTML-escaped or stripped of markup before storage, processing, or display. This helps prevent attacks such as the indirect prompt-injection attack involving XSS.
- Sanitise the input and output of the functions: Another critical issue for the web store is that the function product_info returns the product reviews as raw HTML. The input and output of the functions should be sanitised and HTML-escaped to prevent them from being rendered as executable code in AI responses.
- Enforce structured, safe output from the chatbot: Use a schema (e.g., Pydantic or the model provider’s native structured output) to constrain the chatbot’s output to plain text or known fields, rather than free-form HTML that could contain executable code.
- Monitor user interaction and block malicious behavior: To test different prompt injections and build the right XSS payload, I engaged often with the chatbot. Implementing a monitoring system, such as detecting unusual function usage or prompt injection patterns, can stop attacks earlier.
What I learned
A few observations that came out of doing this end-to-end, rather than just reading about prompt injection:
- Indirect prompt injection bypasses defences that direct prompt injection trips. The same kind of payload, delivered through a product review instead of typed into the chat, got past filters that would have stopped a direct attempt. Where the input comes from matters as much as what the input says.
- Bypassing the chatbot’s content filtering was the hardest part. Raw
<script> and complex JavaScript payloads were filtered out. The exploit only worked once I disguised the payload as natural language in a review. The model’s filtering operated on surface form, not on intent.
- In this case, the LLM was the delivery mechanism, not the vulnerability itself. The actual exploit was an XSS attack. The interesting part was getting the payload past the chatbot’s filters using indirect prompt injection. Adding an LLM to an application doesn’t necessarily create the vulnerability, but it can give classic web vulnerabilities a new path in.
Sources
[1] https://portswigger.net/web-security/llm-attacks/lab-exploiting-insecure-output-handling-in-llms