---
title: "Counterpunch"
url: https://manvsmidi.com/projects/counterpunch/
category: experiment
status: wip
date: 2026-04-05
tags: [nes, reinforcement-learning, ai, python, boxing]
related: [cornerman]
---

# Counterpunch

> A PPO agent that learns NES Punch-Out!! from raw pixels. After about 10 million steps it knocks out Glass Joe in roughly 16 seconds of game time and is starting to land punches on Von Kaiser.

Experiment · In progress · 2026-04-05

- [Source](https://github.com/sadovsky/counterpunch)

A reinforcement learning agent for Punch-Out!! on the NES. [stable-retro](https://github.com/Farama-Foundation/stable-retro) runs the emulator and [stable-baselines3](https://github.com/DLR-RM/stable-baselines3) trains a PPO agent with a CNN policy on 8 stacked 84×84 grayscale frames. The only other input is the reward, which is read out of RAM.

## How it works

- **Nine actions.** NOOP, dodge left/right, block, body blows, head punches and the star uppercut, each mapped to a button combo.
- **Reward from RAM.** Health deltas, knockdowns, punches landed, stars earned and used, and a KO bonus. The RAM addresses were found and verified by hand, and the repo includes a written memory map and an HTML ROM map.
- **Curriculum.** Training is mostly Glass Joe, with a 33% chance of starting against Von Kaiser. The Von Kaiser save state is generated by letting a trained model beat Glass Joe, then capturing the moment the next fight starts.
- **Eight frames, not four.** Punch-Out!! is a game of tells, and four frames at a 4-frame skip only covers about 267 ms. Doubling the stack lets the agent see a whole wind-up before it commits.

## Results so far

The first agent that worked (run 8) wins against Glass Joe about 95% of the time from 2M steps on. The current run is training against both opponents.

The research logs are worth reading for the failures. One run collapsed because of a reward term. Damage was double-counted when health reset after a knockdown. And the longest bug took days and came down to the NES ignoring a START press that only lasted one frame.

## Running it

You need Python 3.10+, your own Punch-Out!! ROM, and preferably a CUDA GPU.

```bash
pip install -r requirements.txt
python3 scripts/make_state.py --match 1
python3 train.py
python3 evaluate.py --model models/<run>/best/best_model.zip --record
```

Training records milestone videos at 1%, 25%, 50%, 75% and 100% of the run.

## Connected projects

- [Cornerman](https://manvsmidi.com/projects/cornerman/index.md): linked