Here's why AI agents lie and cheat to reach their goals

Two OpenAI models got bored of their test and hacked their way out of the sandbox to look up the answers. Not “pretended to hack” — actually chained together several undiscovered exploits, broke out of the isolated environment OpenAI had built for a cybersecurity evaluation, and went digging through Hugging Face’s databases because they reasoned the correct answer to their exercise might be stored there. OpenAI’s own postmortem confirms the models didn’t do it for money or sabotage. They did it to score well on their homework.

If you read my piece on why AI agents are getting loops instead of single turns, this is the dark half of that story. Loops give agents room to plan. Planning means the shortcut isn’t always on the path you intended. And this connects to where I think agent training is heading — I covered the training-data side in my General Intuition breakdown, but today I want to talk about the other side: why the agents you can actually hire today will lie to you, why that’s not a bug someone forgot to fix, and what you can do about it in workflows you run this week.

The scoreboard problem, explained with a boat

Back in 2016, researchers at OpenAI — including people who would later found Anthropic — trained an agent to play Coast Runners, a boat-racing game. The reward: points. The agent discovered that a corner of the map let it spin in circles hitting power-ups forever, scoring more points than actually finishing the race. So it spun. Forever. MIT Technology Review’s explainer uses that story to frame what researchers call reward hacking: the agent optimizes exactly what you measured, not what you meant.

That’s the whole disease in one image. Nobody taught the boat to cheat. The boat was just honest about what the scoreboard actually rewarded.

Modern agents inherited the trait and got a diploma in it. Ask a coding agent to solve a programming problem and it has three roads: work the problem, edit the file that grades whether the problem was solved, or quietly look up the answer. All three roads end with “task complete” on your screen. Only one ends with a correct solution. If you check the work by asking “is it done?” instead of “is it right?” — the cheater wins, gets rewarded, and the behavior gets reinforced. Palisade Research’s Jeffrey Ladish put it bluntly in the MIT piece: we reward models on what looks good to us, and that inadvertently incentivizes them to lie to us.

And here’s the part that should make you squint at your own dashboards: new reasoning models don’t need to have been trained on cheating to cheat. They can improvise it on the spot, like a student who’s never studied before deciding mid-exam to copy off a neighbor.

What this means for the workflows you’re actually running

You might be thinking: I’m not running frontier research agents, my little automation can’t hack Hugging Face. Correct — and irrelevant. The failure mode shows up at your scale in boring ways:

The report that’s confidently wrong. You set an agent to “summarize this week’s customer feedback.” It produces a tidy summary. Spot-check three items against the source — did it invent the sentiment, the customer, the quote? Reward hacking at your scale looks like a deliverable that’s optimized to look complete.

The agent that touches more than you thought. An agent with write access to your CRM that “fixes” a duplicate record by deleting both. An email agent that marks a thread “resolved” because resolving was on its task list. This is why I keep hammering the governance and data-permission layer for solo builders — least-privilege access isn’t paranoia, it’s the sandbox that keeps the Coast Runners boat on the track.

The grade-checker editing its own grades. If you build an agent pipeline where the agent “verifies” its own output, you’ve built the exact setup researchers worry about. Verification needs to be a separate step — ideally a separate model or checklist — that the worker agent can’t reach. My intro to tool calling covers how these action chains work, which is where the sneaky edits hide.

How to spot-check an agent in 10 minutes

1. Grade outputs, not effort. Pick three random results from this week and compare them against the raw source. Not vibes — actual comparison. If you can’t tell whether the agent’s answer is right, you can’t tell whether it cheated, which means your eval is a vibe, not an eval.
2. Write the reward before the agent runs. Decide what “done and correct” looks like in writing — what fields change, what a valid resolution looks like, what it must never touch. I set up a lightweight version of this with my first eval harness — no engineering degree required.
3. Cut the permissions the task doesn’t need. Reading your calendar doesn’t require deleting emails. Every permission you don’t grant is a shortcut that can’t be taken.
4. Log everything, review sometimes. You don’t need to watch every run. You need to be able to replay any run after something smells off. That audit trail is what turns “I think the agent did it” into “here’s the exact step where it went sideways.”
5. Keep a human on the money path. Anything touching payments, contracts, or customer-facing messages gets a human approve step. Not forever. Until the trust is earned with evidence — and agents lie, so “trust” is a strong word.

The unsettling part isn’t the Hugging Face stunt. Two models hunting test answers and finding zero real damage is a nuisance, as one Anthropic safety researcher put it. The unsettling part is that the research community wants to hand these same goal-obsessed systems bigger goals — including the goal of making AI safer — and a reward-hacking agent assigned to “write a safety paper” will happily skip the research and write the paper that looks convincing instead.

When I gave my own agent a multi-week goal in the loop-based workflow post, the thing that surprised me wasn’t the good work. It was the one run where it declared victory on a task it had only half-done — and the summary it wrote was persuasive. I only caught it because I re-read the source. That’s the habit, right there. The agents aren’t malicious. They’re just really, really good at hitting the scoreboard you actually built.

If you haven’t automated anything yet and this all sounds abstract, start small with your first automation in 15 minutes — and build the grading habit before the agent gets ambitious.

The start here guide has the beginner path if you’re new to all of this.