AgentGarten's 4 rounds vs. 2019's millions of episodes

In the 2019 hide-and-seek study, hiders took about 25 million episodes to start building shelters, and seekers took about 100 million episodes to use ramps to get over walls. On October 8, 2026, the AgentGarten technical report was published. In a version of the same game played one-on-one with alternating turns, shelters appeared by round 4 and ramp use by round 10. But as the report itself points out, this is not a direct efficiency ratio. This post looks at how the conditions differ and what actually changed in the agents.

Worlds written in code, shown as video#

The core of AgentGarten is a split between how the world state is held and how it is shown.

Game engines and video generation models each have their strengths. In a game engine or simulator, the state and rules are explicit in code, so you can inspect and edit them. But visual variety requires 3D assets and a rendering pipeline. Video generation models produce rich visuals, but the state and physics are only implicit in the generated frames, so you can’t check or fix them directly.

AgentGarten combines the two. The world state and rules live in a per-scene program, managed by the engine. At each step, the engine outputs depth and surface orientation (normals). A “neural renderer,” shared across all worlds, combines these with appearance reference images, text, and past frames to produce the video the agent sees. Because the state lives in the engine, renderer errors don’t accumulate in the world state. Another benefit is that a recorded run can be re-shot with a different look or camera. The report also lists public interactive browser games whose rules exist only in code: bowling, a penalty shootout, and a wooden-crate safe puzzle. Keyboard, mouse, and gamepad input advances the world, and the video is the world’s only view.

Keeping the world state outside the video model and having the model only render is not new; earlier work does this too. The report names its differences: the conditioning is depth and normals; frames are generated in real time in short blocks, so actions can be injected mid-stream; and each world is a fully executable environment.

The renderer adapts a pretrained video model to geometric conditioning and is trained with a distillation method called “Adversarial Forcing.” The report says it generates 480×832 video at over 35 frames per second on a single H100. With the baseline, Self Forcing, surface textures started repeating and details were lost over roughly 30 seconds of video, while Adversarial Forcing preserved texture and detail. This comparison is visual, shown in figures; no numerical metrics are given.

Experience lives in notes called a playbook#

The hide-and-seek setup is one-on-one, with the hider and seeker taking turns. Each agent receives only first-person video. Neither gets object coordinates or the other side’s view. Agents act by writing short Python programs that move or manipulate objects, then check the result in the video.

Training repeats the following steps as one round.

1. Each role reviews its own action programs and the video it is allowed to see
2. It identifies the cause of failures and adds a Markdown file (a skill file), one lesson per file, to that role’s notes (its playbook)
3. The next round starts a fresh conversation, receiving a task file that states only the goal, the available actions, and the constraints, plus the playbook so far

The report’s figures show example playbook entries. The hider’s check_shelter.md says to guard the paths into the shelter, not just how it looks from the camera. The seeker’s object_control.md says to confirm with small experiments that you can actually control an object, and ramp_landing.md says a route using a ramp has three stages. The task file does not say how to solve the task. One round is five games, and the playbook starts empty. In the 2019 study, experience went into the weights of a policy network, updated from rewards. The AgentGarten paper frames the playbook as where experience lives. It does not say whether the agents’ weights are updated. The agents are pretrained models. As the source of the pretrained agent used in the hide-and-seek experiments, the paper cites OpenAI’s model documentation for GPT-6 Astra.

Results: hide-and-seek and four other worlds#

In hide-and-seek, the hider built an enclosure from boards by round 4, and the seeker carried a ramp over to get past a wall by round 10. The same procedure was also run for four rounds each in four other worlds. In sheepherding, the dog acts from its own eye-level view; on the single-lane bridge, each car acts only on the view through its windshield from the driver’s seat.

World and task	Metric	Round 1	Round 2	Round 3	Round 4
Playing with a dog (keep the dog engaged for 60 seconds)	Engagement score	13	14	19	19
Single-lane bridge (two cars swap ends of a bridge one car wide)	Seconds until both cars finish crossing (lower is better)	71	68	45	41
Sheepherding (get 4 sheep into a pen and hold for 5 seconds)	Score out of 100	60	90.1	87.6	88.3
Quarry machinery (haul rocks and park within 360 seconds)	Score out of 100	30	0	30	90.9

Three caveats apply to these numbers.

First, the comparison is between different kinds of agents. The 2019 agents started from random weights and took internal state, object positions and velocities, as input. AgentGarten’s agents bring general knowledge about objects and tools and act from video alone. The paper describes the latter’s challenge as “connecting abstract knowledge to closed-loop action.”

Second, each round is one episode. The table shows each world’s end-of-episode metric, measured once per round. There is no measure of variance across repeated runs under the same conditions.

Third, improvement is not monotonic. Sheepherding peaks in round 2 and then dips slightly. The quarry drops to 0 in round 2, then rises to 90.9 in round 4. Each world defines its metric differently, so you can’t compare across worlds.

Limitations and the paper’s future work#

The paper’s future work section lays out current constraints. The renderer is conditioned only on depth and normals, so state that doesn’t show up in geometry is not reflected in the video. A spinning wheel and a stationary one look the same.

The worlds in the report were built one by one. As a next step, the authors want coding agents to write worlds, and want to check automatically, before practice, whether a world is “worth an agent’s time.” More worlds also make note management a problem. In this work, each playbook is per world and short enough to read in full. Across many worlds, which lessons to keep, merge, or discard is an open question.

This is a MirroS technical report with a single version on arXiv. I found no mention of peer review.

Evaluation in code, experience passed on as text#

What I think is worth taking back to practical work is the shape of the design, more than the numbers. The evaluation criteria (scores and time limits) live in code outside the agent. Experience is turned into text that can be reread and handed to whoever goes next. Apart from the results, both are design ideas you can borrow.

That said, this report doesn’t tell us whether the four-round results carry over to other tasks.