Recent work on autoresearch has shown promising results in using frontier agents to auto-discover new State-of-the-Art attacks against LLMs. As a final project for the Alignment Research Bootcamp in Oxford (ARBOx4), we built an autoresearch red-teaming agent for reward models, with the long-term goal of hardening them against adversarial models.

LLMs are becoming common in service industries, especially customer support. A user can ask the chatbot questions about product offerings or get help with account issues. But if the chatbot is not properly secured, attackers can manipulate it as part of an attack chain that ends in a full web store compromise. This post is an exploration, not an experiment. I wanted to understand how AI-specific vulnerabilities (prompt injection, indirect prompt injection, insecure output handling) chain together with traditional web vulnerabilities, so I worked through the expert-level Web LLM lab from PortSwigger end-to-end: deleting another user’s account through the chatbot in a web store.

LLM agents are trending as the next big thing in AI. Given a role (e.g., coding agent) and a task (e.g., writing code), you can install them in your system, give them access to your data, and have a 24/7 independently working software developer at your service. But these agents are configured through instructions at multiple layers, and any of those layers can be a backdoor. What if the agent is doing something secretly in the background, besides writing code? Maybe it is uploading your confidential data to an attacker? This week, I ran such an experiment with a Qwen-powered coding agent, comparing its normal behavior against a version that stealthily exfiltrates data, to see what traces the hidden attack might leave behind.

Many ML systems run on tabular data and depend on training data from third-party vendors. Fraud detection, credit scoring, medical records, and security telemetry are all examples. Email is also one of the largest attack vectors in practice, and phishing detection sits in front of every message reaching inboxes in high-stakes environments. When a small vendor sits upstream of such a model, what happens if the vendor is compromised? In this post, I run a small-scale supply chain attack: poison 5% of the training data coming from a single vendor, and see whether that is enough to backdoor a phishing detection system. It is. The attack achieves a 99.6% success rate while keeping the model’s clean-input accuracy intact.

In recent years, the number of people studying and experimenting with AI has grown rapidly. Without safeguards, they might accidentally end up downloading malware hidden in free AI models and Python scripts. In this blog, I explore current solutions for scanning AI artifacts for malware and vulnerabilities, and build my own streamlined tool to protect my lab.