Superhuman Generals.io agent from self-play reinforcement learning

A detailed walkthrough of the development, intended for anyone setting out to build something similar 🤗.

Generals.io¹ is a real-time strategy game with a simple premise. You start as a single crown on a grid, grow an army, spread across the map, and try to capture your opponent's general before they capture yours. The rules take about a minute to learn, but much like Go, that small rulebook unfolds into enormous complexity: games grow into board-spanning fights where a small early advantage can compound for hundreds of turns.

That depth has kept a community around for years — thousands of players, active 1v1 and team formats, regular tournaments, and bots going back nearly a decade.

Human.exe

Human.exe² is a strong bot that has beaten many players, including some with hundreds or thousands of games. It is an impressive piece of hand-engineering (created before the vibecoding era, wow): a strict priority cascade of roughly thirty precomputed plans, a belief model that tracks enemy armies through the fog, and classical optimizers (prize-collecting Steiner trees, min-cost flow, a minimax for short-range fights and capture/defend races). For years it was the bot to beat.

It represents a particular philosophy: encode human knowledge, carefully, by hand. Which runs straight into one of the most quoted ideas in modern AI — Rich Sutton's bitter lesson³: in the long run, general methods that scale with computation tend to overtake systems built from hand-crafted human knowledge.

So there was an obvious bet to make. A learning agent, given enough compute and the right training loop, should be able to surpass even a bot as carefully built as Human.exe — without being told a single human rule. So building one became my master's thesis.

Round one was my master's thesis, built in two stages: supervised pretraining to imitate human play, then reinforcement learning to improve on it through self-play. The idea was to get an agent that plays at all, then push it further.

A few of the agent's games to set the scene. Swipe through the gallery.

‹

Snowballing: once the agent has even a small lead, it balances expanding and attacking, slowly pushing the advantage until it wins. Full game.

Sharp maneuvering: the round-one agent learned precise movements that outplayed opponents game after game. Here red tries to defend both castles, not realizing blue can simply dodge the defense and go for the kill. Full game.

Sharp feint: the agent fakes out the defender — it moves right, red follows to match, then doubles back for the kill, leaving red puzzled. Full game.

Backdooring: the agent learned to keep a stack of army hidden in the fog and strike from behind. It most likely comes from its own limited memory — in self-play it learned that the opponent, like itself, won't keep track of an army parked in the fog. That happens to match how people forget too, so the agent pulls the trick off again and again. This clip is shown from red's point of view: from there, the attack barely shows up before it lands. Full game.

Another backdoor: same trick, another game — again from red's point of view, the kill comes out of the fog. Full game.

Failure mode: the agent never fully learned to measure distances between cells. Here it tries to move “through the wall” to minimize Manhattan distance, not realizing it has to go around. Full game.

›

Stage one: learning from humans

Dataset

Training an agent from scratch on a small compute budget is hard. At the start it moves at random, and on a full-size board it almost never captures the enemy general, so the win/loss signal it needs to learn from almost never appears. Supervised pretraining avoids this by starting from human play instead. Generals.io keeps match replays on its servers, and 347,000 of them were downloaded as the raw dataset; filtering then removed the degenerate games and kept only those involving strong players, leaving 16,320 games, about 9 million moves. The cleaned dataset is on 🤗 Hugging Face.

Network architecture

The network's input is the board encoded as a stack of feature planes — terrain, cell ownership, army counts, and the fog-of-war mask — together with memory planes that summarize the game so far: the last known positions of revealed generals and castles, which cells have been explored, which cells the opponent has seen, and the recent moves of both players. Under fog of war most of the board is hidden on any given frame, so these memory planes give the network a compact view of the history it would otherwise have to learn using memory modules such as an RNN.

The architecture is U-Net style, taken from DeepNash⁴, which used it for Stratego. A convolutional U-Net torso turns the stacked observation into a board embedding that feeds two heads, one for the policy and one for the value; the implementation is on GitHub.

Training objective

The action space is defined per cell: for every cell the agent can pass, or move all or half of its units in one of the four directions — an H×W×9 grid of possible moves. Pretraining optimizes both heads at once. The policy head is trained by cross-entropy to predict the move the human actually played at each step — this is the part that matters for behavior cloning. The value head, trained to predict which player eventually won the game, isn't needed to imitate moves at all; it is trained because stage two is actor-critic, and a critic warm-started on human play is a far better initialization than a random one.

Trained this way, the pretrained network already played like a competent human — strong enough to reach roughly the top 150 players on the ladder, a 61 ⭐ rating. That was the starting point for stage two.

Stage two: reinforcement learning

Behavior cloning extracts what it can from human games, but it can only imitate — the policy never surpasses the players it learned from. Going further means producing new data rather than replaying old games, which is what reinforcement learning is for: the agent plays, keeps what works, and improves from its own results.

Training is self-play: the agent's opponents are frozen copies of earlier versions of itself — beat the snapshot, become the snapshot, repeat.

Simulator

Those games have to be played somewhere, so a custom Generals.io simulator was built. Because reinforcement learning is data-hungry, the environment is vectorized to run many games in parallel, and it exposes a gym-like interface, so the training code looks like any other RL setup.

The simulator ran on CPU, and that was the bottleneck rather than the GPU: training went only as fast as games could be produced. To get more out of each iteration, only the policy and value heads were fine-tuned while the U-Net torso stayed frozen; updating the whole network barely helped at this stage and was much slower.

The training objective

The agent is trained with PPO⁵, an actor-critic algorithm. It learns two things side by side: a policy (the actor) that picks the moves, and a value function (the critic) that estimates how well the agent can expect to do from a given state.

It plays, and for every move it compares how things actually turned out against what the critic expected. Moves that did better than expected get increased probability, moves that did worse get decreased probability. PPO adds one safeguard: a single update can't drift the new policy too far from the previous one, so the agent improves in small, safe steps.

The RL phase already improved on the behavior-cloning agent, but it always converged to a single style: extremely aggressive, mostly gathering army and attacking. Against humans this worked surprisingly well — it climbed from around the top 150 to the top 100 of the ladder — but players who knew how to defend and build a lasting advantage could beat it.

Stage three: patching a local optimum

That aggressive style wasn't the only thing the agent could have learned — it was a local optimum, and on the compute budget of round one, a hard one to escape. Aggression works well enough that the gradient keeps pointing back at it, and without enough exploration to find the slower, more positional lines, the agent never climbs out. One way out is more compute: explore wider and longer until the better optimum surfaces. The cheaper way is to patch the objective and nudge the agent out of the basin by hand. Round one took the cheaper route, with two patches on top of plain self-play: reward shaping and a poor man's fictitious play, each covered below.

Reward shaping

In plain self-play the reward is very sparse: just \(\pm 1\) at the end of the game. With \(\gamma = 1\)?Discount factor γIt sets how much future reward counts: rewards add up as r₁ + γ r₂ + γ² r₃ + …, so γ < 1 makes distant reward worth less. At γ = 1 every step counts the same. that means every action in a game gets the exact same credit, even though they didn't contribute equally — a winning game can still contain bad moves. Working out which actions actually deserve the credit is the famous credit assignment problem.

One way to help is to not give reward only at the end of the game, but after every action, based on how good that action seems. The risk is that hand-crafted reward functions often get exploited by the learning algorithm and don't yield the behavior you wanted. But one form of shaping avoids that — potential-based reward shaping⁷ (PBRS). It assigns a potential \(\phi(s)\) to every state, and on a move from \(s\) to \(s'\) it adds an extra reward equal to the change in potential:

So the original reward is augmented by the difference of potentials: move to a higher-potential state and the action gets rewarded more, move to a lower one and it gets penalized. In our case the potential is a weighted sum of log-ratios?Why a log ratio?It's symmetric around parity — twice the opponent's army reads as +1, half as −1. A raw ratio would be skewed: +2 versus −½. between the agent's and the opponent's land, army, and castle counts:

In plain terms, the potential \(\phi\) is high in states where the agent is materially ahead (more land, army, and castles than the opponent) and low when it's behind.

Potential-based shaping comes with a guarantee that ad-hoc reward bonuses don't: it leaves the set of optimal policies unchanged. The dense signal changes how fast the agent learns, not what it is ultimately trying to do.

That guarantee is weaker in practice than it sounds: it describes the limit, not the finite-compute path, and runs with different \(c_1, c_2, c_3\) converged to visibly different players — one expands harder, another hoards army, another prioritizes castles. The improvement itself, though, was strong and consistent across all of them: every shaped variant beat the unshaped agent comfortably, expanding territory far more effectively and sometimes investing in capturing a castle — something the attack-everything version never did.

A poor man's fictitious play

Training only against the latest copy of itself, the agent can overfit to that single opponent and fall into non-transitive cycles rather than improving steadily. A pool of past versions instead pressures it toward play that is robust across opponents.

So the agent instead trains against a small pool of its own checkpoints. The pool is kept at three agents, each with its own style; every game is played against one of the three, drawn at random. Once the learner's win rate against the pool passes 60%, a fresh snapshot of it joins and the oldest agent drops out, and training continues against the updated pool. It is a poor man's fictitious play⁸: a handful of checkpoints rather than full neural fictitious self-play (NFSP)¹², but enough to make it robust to more than one style of play. Round two drops it entirely, relying on MMD's convergence properties instead of a hand-built pool.

Where round one landed

Behavior cloning, then shaped self-play against the checkpoint pool, cleared the wall — a learning agent above the strongest hand-engineered bot, with no human rules anywhere in it.

Most of that climb was likely reward shaping alone, not the pool on top. The Human.exe edge is real (95% CI 50.6–59.0%); round two closes the gap to the very best humans.

After the thesis wrapped up, we counted it a success — but it still bothered me that round one couldn't be pushed any further. So I looked back and saw that its limits had a recurring shape. Supervised pretraining existed because a from-scratch agent on a slow simulator would almost never stumble into a win. Reward shaping existed because the true win/loss signal was too sparse to learn from on the samples available. Both were patches on the same deeper problem: the training loop couldn't generate enough experience, so ad-hoc tricks were used to squeeze the most out of what little there was.

So for round two I stopped bandaging symptoms and went after the sickness itself: the experience bottleneck. Rather than work around a starved training loop, the whole stack was rewritten in JAX to maximize simulator throughput and built on architectures that scale with data, so experience stopped being the constraint. With the bottleneck gone, the patches had nothing left to do, and every one of them went:

• No behavior cloning. The agent never sees a human game.
• No reward shaping. Sparse win/loss reward only.
• No population, no pool. A single policy, plain self-play.

What replaced the patches was speed and a few targeted choices: the JAX simulator at ~50M steps/s, a transformer policy, a parameter EMA at deployment, and advantage filtering. This agent reached #1 on the ladder and beat the top humans head-to-head. The sections below cover each piece in turn.

A simulator 10,000× faster

The first critical piece was rewriting the environment in JAX. Round one's simulator ran on CPU — 64 cores, 128 parallel games, roughly 3,500 steps per second, with the GPU mostly waiting for rollouts. Done right, a JAX rewrite moves the entire game onto the accelerator: board state lives in arrays, the rules become vectorized array operations, and tens of thousands of games step in lockstep as one program. On a single GPU, throughput went from ~3,500 steps per second to ~50 million — a four-orders-of-magnitude jump.

Samples that cheap change what's necessary. There's no need to bootstrap from human games when the agent can play billions of its own, and no need to densify the reward when the sparse win/loss signal arrives millions of times per run. The patches from round one didn't have to be carefully replaced — they simply had nothing left to patch.

The network

The second change was the architecture. Round one's U-Net was inherited from DeepNash; round two swaps it for a transformer, adapted from another Stratego agent, AtaraxosAI/stratego. Two things make it a better fit for round two: transformers are known to scale better with large amounts of data, the regime the rewrite created, and their self-attention is global, so the agent can reason across long distances on the map in a single step rather than passing information through the U-Net bottleneck. The board is sliced into 3×3 patches and each patch is projected into a token; two extra tokens carry the opponent's land and army totals over time, the scoreboard information every player watches to reason through the fog. A transformer torso processes the sequence, and the value head and the familiar H×W×9 policy head sit on top.

under the hood Choosing the shape of the network ▶

The question was where to spend the network's capacity: a deeper network with smaller embeddings, or a shallower one with larger embeddings. To compare them fairly, the parameter count was fixed across three setups (wider, medium, and deeper), each trained on a smaller version of the game.

On the 15×15 board, the wide network consistently reached the highest performance given the fixed time budget. It also had the most throughput — about twice the deep network's at the same parameter count — so it ran roughly 2× more iterations in the same wall-clock. Depth seemed likely to matter more as the board grew, though, and the medium network wasn't lagging far behind, so the medium shape won out.

The medium configuration was then scaled up to many more parameters, keeping its depth-to-width proportion. That scaled-up network is the one deployed:

Parameter	Value
depth	7
embed_dim	448
n_head	8
ff_factor	3
patch_size	3

All in, the deployed network is 15.4M parameters. Half that size might well have done as well or better, but this was a single “yolo” run, and it was good enough.

Cherries on top

The remaining pieces were ablated to see which actually matter; a few of them below.

Reward shaping

Round one used potential-based reward shaping, and it helped — probably because, from the supervised checkpoint, RL rushed toward a hyper-aggressive style, and shaping steered the agent toward material advantage. Throughput was also ~100× lower, so a denser reward got more out of each game.

Round two flipped that. With far more data, the win/loss signal alone tells the agent enough about what wins to make shaping unnecessary — and any shaping not completely aligned with winning only biases it towards suboptimal behavior. In the ablation, shaped reward improves slower early, then degrades and destabilizes.

Parameter EMA

At deployment, the network's weights are an exponential moving average of the training iterates rather than the latest checkpoint. The agent is strong without it, but even across 7 days of training on 4×H200s, the EMA checkpoint kept a higher Elo than the last iterate (about 30 Elo by the end of the run). For a one-line change costing essentially zero compute, that's a nice improvement.

Advantage filtering

Many games are decided long before they end — positions where a person would already resign, but the agents play on anyway. If the game then ends as expected, those late samples carry little new information, so there's little point training on them.

So training updates keep only the top 25% of observation–action pairs, ranked by predicted advantage, and drop the rest — which also makes the update step much faster. The trick comes from Sokota et al.'s Stratego work¹³, where keeping only the highest-advantage transitions saved a lot of wall-clock and was more sample-efficient; it helped here too.

This was ablated on a smaller version of the game, on a single GPU. A smaller fraction gives higher variance across seeds, but also the best single seed. Not every setting was run at full scale, but on the 4-GPU runs the top 25% worked very well.

Hyperparameters

For the tuners: the main hyperparameters, their values, and — more usefully — how much getting each one right seems to matter.

Honestly, there are too many hyperparameters to ablate properly, and most of the values above are educated guesses or defaults from the literature. The ones that got real attention were the network architecture, then the learning rate and GAE λ. In hindsight, minibatch size deserved more experimentation — left near a default here, it keeps turning up as a surprisingly important factor across RL work, and it's the first thing worth tuning harder next time.

A head that learned to see through the fog

Most of the network is distributed mush under a microscope, but one attention head (value token attending board patches, layer 6, head 4) turns out to encode a belief over where the enemy general is hiding — and it reasons about it very well. It is strikingly accurate, often pinpointing the hidden enemy base within a handful of moves.

‹

Game Attention

Left: one self-play game. Right: the same game, colored by where the head attends — its belief over the hidden enemy general (value token attending board patches, layer 6, head 4).

Game Attention

A different game (another seed). The same head does the same job — a belief over the hidden general that sharpens and snaps onto it once it is seen.

Game Attention

And once more, on a third game — the same tracking behavior, not a one-off.

›

A few heads seemed to switch jobs partway through a game, as if flipped by some event. One that spent the early game tracking castles changed behavior completely the first time the enemy general was revealed — from then on it attended to something else entirely.

Top of the ladder

The agent was evaluated on the public 1v1 ladder. In its first thousand rated games, played under a few plain nicknames (Average Joe among them), it won 815 and finished as the highest-rated player on the leaderboard — above every human and every other bot.

At the time of writing, the highest-ranked human on the live ladder is shimatetsu at 101, while Average Joe sits at 118.

A leaderboard rating averages over the whole population, so the head-to-head records against individual top humans are worth a separate look. Each was a normal rated game from the matchmaker, not an exhibition, and both players knew the opponent was a bot.

Against the two highest-ranked humans alone the agent went 199–70 across 269 games — significant at p < 0.001 against a coin flip. And the two bots that mattered, Human.exe and its own round-one predecessor, were swept 40–0.

The agent in action

A few replays of the agent playing, each picked to show a specific behavior. Swipe through the gallery. More games can be watched on the agent's ladder accounts, Average Joe and L_7d_gae90_30k_ema.

‹

The agent shows perfect maneuvering: during the attack it splits its army into three shards, confusing the enemy and forcing a mistake that converts into a win. Full game.

Continual suffocation: against Human.exe, the agent encircles it first and forces it to defend. If Human.exe attacks, it gets backdoored, so it has a hard time. The whole game, sped up 10×. Full game.

Relentless attacking: the agent has a finely calibrated sense of when to focus on its own expansion and when to simply attack the enemy's resources and snowball the lead. Across the whole game (sped up 10×) it continually attacks the opponent's cities, steadily building a material advantage. Full game.

Perfect multitasking: the agent splits its army into offense and defense and interleaves their movement. One half captures an enemy city while the other holds off a lethal attack, leaving it in a completely winning position. Full game.

The agent has nice spacing: when attacking, it cleverly sidesteps the enemy army, and once the enemy is gone it captures land near the enemy base, giving important exploration and presence. Full game.

›

Closing remarks

At heart, this project is one more case of the bitter lesson paying off, another instance of letting compute go brr :D The jumps didn't come from clever heuristics but from tools that turn raw computation into progress: a simulator fast enough to flood the agent with experience, and an architecture and training loop that keep improving the more you feed them. With those in place, reaching superhuman play took nothing exotic, just general policy-gradient methods and enough experience to learn from, with no game-specific solver and no handcrafted strategy. It still isn't the best bot the recipe could produce. Training the memory rather than hand-crafting it, trading network size for more games played, or giving the agent an explicit belief model over the enemy's army and general's location would all likely help.

To be clear about the order of events: this was the first agent to play this game at a superhuman level. The simulator and the agent were both published before this post went out, and the gap is already closing — people are building on both, some catching up, some arguably ahead, and as of writing the agent sits second on the ladder rather than first. That was always the point of open-sourcing it.

Mostly, the hope is that this is useful to someone. It tries to show the process and not only the final numbers — the dead ends, the patches, the calls worth making differently — because that is the part that rarely survives into a paper, and it's the part most likely to help another applied-RL practitioner shipping something of their own.

Shoutout to Martin for supervising the thesis, Jonas for hyping me up through the whole project and giving useful feedback, EklipZ for building Human.exe in the first place — beating it was the whole point of the project, so without it none of this would exist — and for help getting connected to the real servers and a lot of useful Generals.io detail, and Juraj for implementing the remote connection.

1. Generals.io. generals.io
2. EklipZ. Human.exe (generals-bot). github.com/EklipZgit/generals-bot
3. R. Sutton. The Bitter Lesson. 2019. incompleteideas.net
4. J. Perolat et al. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 2022. arXiv:2206.15378
5. J. Schulman et al. Proximal Policy Optimization Algorithms. 2017. arXiv:1707.06347
6. J. Schulman et al. High-Dimensional Continuous Control Using Generalized Advantage Estimation. 2015. arXiv:1506.02438
7. A. Ng, D. Harada, S. Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. ICML, 1999. PDF
8. G. W. Brown. Iterative Solution of Games by Fictitious Play. Activity Analysis of Production and Allocation, 1951.
9. M. Lanctot et al. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning. NeurIPS, 2017. arXiv:1711.00832
10. S. Sokota et al. A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Games. ICLR, 2023. arXiv:2206.05825
11. M. Rudolph et al. Reevaluating Policy Gradient Methods for Imperfect-Information Games. 2025. arXiv:2502.08938
12. J. Heinrich, D. Silver. Deep Reinforcement Learning from Self-Play in Imperfect-Information Games. 2016. arXiv:1603.01121
13. S. Sokota et al. Superhuman AI for Stratego Using Self-Play Reinforcement Learning and Test-Time Search. 2025. arXiv:2511.07312
14. M. Zinkevich, M. Johanson, M. Bowling, C. Piccione. Regret Minimization in Games with Incomplete Information. NeurIPS, 2007.

If this was useful in your own work, you can cite the round-two paper:

@misc{straka2026generals,
  title         = {Superhuman AI for Generals.io Using Self-Play Reinforcement Learning},
  author        = {Matej Straka and Viliam Lisý and Martin Schmid},
  year          = {2026},
  eprint        = {2606.23348},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2606.23348},
}

Matej Straka

PhD student at Charles University in Prague, interested in deep reinforcement learning.