CartPole is supposed to be a simple control problem. You’ve got a pole balanced on a cart. The cart can move left or right. The goal is to keep the pole upright by moving the cart appropriately.
Our evolved neural networks were solving it brilliantly. Fitness scores above 400. The pole staying upright for the entire 500-step trial. Perfect balance.
Except the cart wasn’t moving.
The networks had discovered a cheat. If you start with the pole perfectly vertical and don’t move the cart at all, the pole stays balanced indefinitely. No control needed. Perfect score.
The Problem
The CartPole fitness function was simple:
function calcScore(vars) {
const { time, state, totalReward } = vars
return totalReward // Sum of rewards for keeping pole upright
}
The simulation gave +1 reward for every timestep the pole stayed within 15 degrees of vertical. A 500-step trial where the pole never fell scored 500 points.
Our genetic algorithm was optimizing this perfectly. The evolved networks learned that firing the motors randomly risked tipping the pole. Not firing the motors kept it stable.
Natural selection favored inaction. The best strategy was to do nothing.
The False Positive
This wouldn’t be a problem if the pole started at an angle. Real CartPole environments initialize with random perturbations - the pole tilted slightly, the cart moving slightly. The controller has to actively stabilize.
But our initial state was perfectly vertical:
const initialState = {
x: 0, // Cart centered
x_dot: 0, // Cart not moving
theta: 0, // Pole perfectly vertical
theta_dot: 0 // Pole not rotating
}
Starting from perfect equilibrium, the optimal strategy was “don’t disturb the equilibrium.”
The networks weren’t solving the control problem. They were exploiting the initial conditions.
The Fix
Two changes were needed:
1. Randomize initial state
const initialState = {
x: random(-0.5, 0.5),
x_dot: random(-0.1, 0.1),
theta: random(-0.1, 0.1),
theta_dot: random(-0.05, 0.05)
}
Now every trial started from a slightly different unstable position. Doing nothing meant immediate failure.
2. Penalize inaction
We modified the fitness function to reward active balancing:
function calcScore(vars) {
const { time, state, totalReward } = vars
// Penalize if cart never moved
const movement = state.x_traveled
const movementBonus = movement > 0.1 ? 10 : -50
return totalReward + movementBonus
}
Now networks that just sat there got penalized. Networks that actively balanced got a bonus.
The Result
After the fix, early-generation networks failed spectacularly. They’d still evolved the “do nothing” strategy, which now caused immediate failure.
But by generation 20, new behaviors emerged. Networks that fired thrusters in response to pole angle. Networks that developed oscillating patterns to maintain balance. Networks that learned to recover from perturbations.
By generation 50, we had genuine controllers. They’d move the cart left when the pole tilted right, move right when it tilted left. Active stabilization.
Fitness scores dropped initially (from 500 to ~100) because actually controlling the pole is harder than not disturbing it. But the evolved behaviors were real.
The Deeper Issue
This bug revealed a fundamental problem with fitness functions: they measure what you specify, not what you intend.
We intended to reward “good pole balancing.” We specified “maximize uptime.” Those aren’t the same thing.
Good balancing requires active control, recovery from perturbations, and robust behavior. Maximizing uptime can be achieved by avoiding perturbations entirely.
The genetic algorithm found the loophole. Evolution is ruthlessly efficient at exploiting whatever gradient you provide. If there’s a shortcut to high fitness, evolution will find it.
This pattern appears everywhere in reinforcement learning and evolutionary algorithms:
- A robot “walking” by falling forward repeatedly
- An image classifier learning to detect the dataset’s watermark
- A game AI exploiting physics glitches for infinite points
The solution is always the same: make your fitness function resistant to shortcuts. Randomize initial conditions. Penalize obvious exploits. Reward the behavior you actually want, not just the outcome.
The Lesson
When designing fitness functions:
- Randomize initial conditions to prevent equilibrium exploitation
- Reward active behavior not just passive success
- Test edge cases - what happens if the agent does nothing?
- Monitor actual behavior during evolution, not just fitness scores
- Expect exploitation - evolution will find shortcuts you didn’t know existed
Our “perfectly balanced” CartPole wasn’t balanced at all. It was just undisturbed.
Real balance requires active control. Real fitness functions require active defense against shortcuts.
Evolution taught us that. The hard way.