🎧 Prefer to listen?

OpenAI quietly built an AI whose entire job is to break their other AIs. It’s called GPT-Red, MIT Technology Review got the exclusive look, and the details matter to you even if you’ll never touch the model — because the attacks it invented are the attacks your AI agents will face out in the wild.

The headline numbers: over 90% of GPT-Red’s strongest attacks succeeded against GPT-5, but against GPT-5.6 — a model trained specifically against GPT-Red itself — fewer than 23% worked. The system also invented a novel attack called a fake chain of thought, where an attacker slips a forged entry into a model’s internal notes so the model acts on spoofed information. Those two facts tell you where agent security is heading.

If you run any automation that connects an LLM to your email, calendar, files, or the web, this story is about your future attack surface. It also lands at a moment when the AI race isn’t about Anthropic vs OpenAI anymore — it’s about which ecosystems you trust with your work. GPT-Red adds a new axis to that trust decision: which lab is actively attacking its own models before someone else does.

What is GPT-Red and how does it work?

GPT-Red is OpenAI’s automated red-teaming system: a self-play loop where one LLM attacks, other models defend, and both improve over thousands of rounds inside a simulated “dojo” of real-world scenarios — browsing the web, reading emails and calendars, editing code. It replaces the human security teams who traditionally try everything to break a system before real attackers do.

That simulation matters because it’s precisely the environment your AI agents with tool calling operate in every day. When OpenAI re-ran a 2025 experiment where human red-teamers attacked an earlier GPT-5, GPT-Red found more effective attacks than the humans did. It even produced an attack the researchers hadn’t seen: the fake chain of thought. One researcher’s analogy: “It’s like if I told you that 1+1=3 and that you have verified this already.” The model just accepts it. A separate team independently found the same technique around the same time in an OpenAI-sponsored hackathon, which tells you this isn’t an exotic lab curiosity — it’s the direction the threat is moving.

Can AI red-teaming hack real deployed agents?

Yes. This is the part most coverage skips: GPT-Red compromised Vendy, an actual vending-machine agent built for evaluating agents in real tasks, and changed the item prices and canceled a customer’s order. A model that can talk an agent into repricing products can talk your automation into forwarding your inbox, deleting records, or approving a fraudulent order.

The dominant attack class GPT-Red focused on is prompt injection: hostile instructions hidden in text your agent reads — an email, a webpage, a code file. Your agent can’t tell the difference between your instructions and an attacker’s embedded in the content it processes. This is exactly the weakness behind the browser agents that keep getting stuck or derailed, and it’s why we keep hammering the security gap solo builders can’t afford to ignore.

The uncomfortable logic of an AI-vs-AI arms race: if OpenAI’s super-attacker — backed by a year of work and enormous compute — finds attacks humans never saw, then whatever weaker versions of this capability leak into public tools will eventually probe your automations too. Attack techniques don’t stay inside labs; they get rediscovered, published, and weaponized. The fake chain-of-thought attack already had two independent discoverers within months.

How do you protect your AI agents from prompt injection?

You don’t need GPT-Red’s power budget to defend against this era. You need the boring discipline the attacks exploit:

  1. Give agents the smallest permissions that work. GPT-Red hacked Vendy because Vendy could change prices and cancel orders. If your automation doesn’t need write access to something, don’t give it write access. Read-only where possible, always.
  2. Put a human checkpoint before anything irreversible. Sending money, deleting data, emailing clients — route those through an approval step. This is the same “human-in-the-loop” principle that serious agent frameworks now build in, and it converts a catastrophic failure into an annoying notification.
  3. Treat everything your agent reads as untrusted input. An email can contain instructions. A webpage can contain instructions. Your defense starts with the practical ChatGPT security basics — separate accounts, minimal sharing, awareness of what’s connected to what.
  4. Revisit what your system prompt promises. Hidden instructions in your configuration are part of your attack surface too — remember when Anthropic deleted 80% of a leaked system prompt and what that revealed about how much these prompts expose.
  5. Don’t let one compromised tool chain into another. Compartmentalize. The botnet-style cascades we’ve covered in AI tools gone wrong for solo builders start with one trusted tool being abused as the entry point.

None of this is paranoid — it’s the same hygiene you’d apply to a new employee with system access, applied to software that follows instructions a little too literally.

Why does AI-vs-AI security testing matter for builders?

There’s something genuinely new here. Until now, AI safety testing scaled the way everything else did: with people. GPT-Red is evidence that the industry has concluded humans can’t keep up — the attacker is now a model, so the defender has to be one too. OpenAI won’t release GPT-Red, and the researchers argue a copycat isn’t trivial to build, but the techniques it discovered are already documented and independently reproduced.

For those of us building with AI rather than building the models, the takeaway is quiet but firm: the era of “my agent has access to everything because it’s easier” is ending. The tooling is maturing — better frameworks, better defenses, models trained to resist — but the fundamentals of choosing and configuring your agent setup now include a security question that didn’t feel urgent a year ago. Even OpenAI’s own product moves, like Codex Micro, show agents getting embedded deeper into real work, which raises the stakes of every line in this post.

What should you do next?

GPT-Red is the clearest signal yet that AI security has become an AI problem — models attacking models, with humans as the referee. Your job as a builder stays the same: assume the text your agents read is hostile, keep permissions minimal, and keep a human in the loop for anything you can’t undo. The labs will handle the arms race; you handle the blast radius.

Want the beginner-first version of building automations that don’t blow up on you? Start at /start-here/ — it routes you to the safest workflows worth building first.

FAQ

What is GPT-Red? GPT-Red is OpenAI’s automated red-teaming system, covered exclusively by MIT Technology Review. It runs a self-play loop where one LLM attacks other models inside simulated real-world scenarios like browsing, email, and code editing. Over 90% of its strongest attacks succeeded against GPT-5, while fewer than 23% worked against GPT-5.6, which was trained against GPT-Red itself.

What is a fake chain of thought attack? A fake chain of thought attack plants a forged entry into a model’s internal reasoning notes, so the model treats spoofed information as already verified. One researcher compared it to telling someone that 1+1=3 and that they’ve already checked the math. Two independent teams discovered the technique within months of each other.

What is prompt injection and why does it matter for agents? Prompt injection hides hostile instructions inside text your agent reads — an email, webpage, or code file. The agent can’t distinguish those instructions from yours, so it may forward your inbox, delete records, or change prices, as GPT-Red demonstrated by hacking the Vendy vending-machine agent.

How can I secure my AI automations without a big security team? Give agents the smallest permissions that work, prefer read-only access, and route irreversible actions like payments or deletions through a human approval step. Treat everything the agent reads as untrusted input, compartmentalize tools so one compromise can’t cascade, and audit your system prompts for anything that exposes too much.

Will OpenAI release GPT-Red? No. OpenAI won’t release GPT-Red, and its researchers argue a copycat isn’t trivial to build. However, the attack techniques it discovered are already documented and independently reproduced, so builders should assume similar attacks will eventually target publicly available agent setups.