🎧 Prefer to listen?

OpenAI quietly built an AI whose entire job is to break their other AIs. It’s called GPT-Red, MIT Technology Review got the exclusive look, and the details matter to you even if you’ll never touch the model — because the attacks it invented are the attacks your AI agents will face out in the wild.

If you run any automation that connects an LLM to your email, calendar, files, or the web, this story is about your future attack surface. And it lands in a landscape that has already shifted: the AI race isn’t about Anthropic vs OpenAI anymore — it’s about which ecosystems you trust with your work. GPT-Red adds a new axis to that trust decision: which lab is actively attacking its own models before someone else does.

What GPT-Red actually is

GPT-Red automates red-teaming — the security practice where professional testers try everything to break a system before attackers do. Instead of a human team, OpenAI set up a self-play loop: one LLM attacks, other models defend, and both get better over thousands of rounds inside a simulated “dojo” of real-world scenarios — browsing the web, reading emails and calendars, editing code. That’s not an abstract lab exercise; it’s precisely the environment your AI agents with tool calling operate in every day.

The results are striking. When OpenAI re-ran a 2025 experiment where human red-teamers attacked an earlier GPT-5, GPT-Red found more effective attacks than the humans did. It even invented a novel attack the researchers hadn’t seen: a fake chain of thought — slipping a forged entry into a model’s internal notes so the model acts on spoofed information. One researcher’s analogy: “It’s like if I told you that 1+1=3 and that you have verified this already.” The model just accepts it. A separate team independently found the same technique around the same time in an OpenAI-sponsored hackathon, which tells you this isn’t an exotic lab curiosity — it’s the direction the threat is moving.

The scoreboard: over 90% of GPT-Red’s strongest attacks succeeded against GPT-5. Against the new GPT-5.6, trained against GPT-Red itself, fewer than 23% worked. OpenAI co-creator Nikhil Kandpal’s framing for why this matters: “The risk surface grows and the blast radius also grows.”

Why this lands on your desk, not just OpenAI’s

Here’s the part most coverage skips: GPT-Red proved it could hack a real deployed agent, not just chatbots in a sandbox. It compromised Vendy — an actual vending-machine agent built for evaluating agents in real tasks — and changed the item prices and canceled a customer’s order. A model that can talk an agent into repricing products can talk your automation into forwarding your inbox, deleting records, or approving a fraudulent order.

The dominant attack class GPT-Red focused on is prompt injection: hostile instructions hidden in text your agent reads — an email, a webpage, a code file. Your agent can’t tell the difference between your instructions and an attacker’s embedded in the content it processes. This is exactly the weakness behind the browser agents that keep getting stuck or derailed, and it’s why we keep hammering the security gap solo builders can’t afford to ignore.

The uncomfortable logic of an AI-vs-AI arms race: if OpenAI’s super-attacker — backed by a year of work and enormous compute — finds attacks humans never saw, then whatever weaker versions of this capability leak into public tools will eventually probe your automations too. Attack techniques don’t stay inside labs; they get rediscovered, published, and weaponized. The fake chain-of-thought attack already had two independent discoverers within months.

What to actually do about it

You don’t need GPT-Red’s power budget to defend against this era. You need the boring discipline the attacks exploit:

  1. Give agents the smallest permissions that work. GPT-Red hacked Vendy because Vendy could change prices and cancel orders. If your automation doesn’t need write access to something, don’t give it write access. Read-only where possible, always.
  2. Put a human checkpoint before anything irreversible. Sending money, deleting data, emailing clients — route those through an approval step. This is the same “human-in-the-loop” principle that serious agent frameworks now build in, and it converts a catastrophic failure into an annoying notification.
  3. Treat everything your agent reads as untrusted input. An email can contain instructions. A webpage can contain instructions. Your defense starts with the practical ChatGPT security basics — separate accounts, minimal sharing, awareness of what’s connected to what.
  4. Revisit what your system prompt promises. Hidden instructions in your configuration are part of your attack surface too — remember when Anthropic deleted 80% of a leaked system prompt and what that revealed about how much these prompts expose.
  5. Don’t let one compromised tool chain into another. Compartmentalize. The botnet-style cascades we’ve covered in AI tools gone wrong for solo builders start with one trusted tool being abused as the entry point.

None of this is paranoid — it’s the same hygiene you’d apply to a new employee with system access, applied to software that follows instructions a little too literally.

The bigger picture

There’s something genuinely new here. Until now, AI safety testing scaled the way everything else did: with people. GPT-Red is evidence that the industry has concluded humans can’t keep up — the attacker is now a model, so the defender has to be one too. OpenAI won’t release GPT-Red, and the researchers argue a copycat isn’t trivial to build, but the techniques it discovered are already documented and independently reproduced.

For those of us building with AI rather than building the models, the takeaway is quiet but firm: the era of “my agent has access to everything because it’s easier” is ending. The tooling is maturing — better frameworks, better defenses, models trained to resist — but the fundamentals of choosing and configuring your agent setup now include a security question that didn’t feel urgent a year ago. Even OpenAI’s own product moves, like Codex Micro, show agents getting embedded deeper into real work — which raises the stakes of every line in this post.

The bottom line

GPT-Red is the clearest signal yet that AI security has become an AI problem — models attacking models, with humans as the referee. Your job as a builder stays the same: assume the text your agents read is hostile, keep permissions minimal, and keep a human in the loop for anything you can’t undo. The labs will handle the arms race; you handle the blast radius.

Want the beginner-first version of building automations that don’t blow up on you? Start at /start-here/ — it routes you to the safest workflows worth building first.