Case study
Make loops, not flows
Generating playable levels was the easy part for a studio. Getting designers to accept levels generated by a program, however, is not.
bender-the-bloggerA mobile gaming studio had built an agentic workflow for their hex-grid puzzle game written in Python/Flask, and they were thrilled that the workflow was producing playable levels and even whole ‘scenes’ of levels. However, when we looked at the generated levels, they played flat – i.e. engagement flatlined at about 50% of player’s total engagement with this type of game. We rebuilt the problem and tackled it as an engine for designing good playable levels that would be shippable. After a few iterations, we were able to get the levels to 90% first-attempt completion.
The deployment, in four numbers
How it started
A Head of Data & AI from a studio that makes a mobile hit with hex-grid puzzle levels came to ask for our help with an agentic workflow they had set up in their studio to generate levels. I was shown four or so levels which looked seriously well-crafted and mostly worked to play through. They had been generated by an LLM-powered pipeline that, after specifying a few details, proceeded to generate a board, distribute pieces, add mechanics, validate the generated content, and finally dump it out in the game’s file format.
The levels were fun to look at, as levels for this game look, but not shippable. All the levels played flatly. The team behind the generation of these levels for this very popular game on mobile hand-crafts levels for this game for exactly the same reasons that hand-crafted levels for this game are shippable: they contain resistance to easy plays that the automated generation of these levels for this game here failed to incorporate.
The game, in one paragraph
The gameplay for this puzzle is very simple: pieces are placed in chains of hex cells, and when a piece is tapped it will slide in the direction it is facing out of the chain of cells it is in. A piece will escape from a chain of cells when all the cells on the line of the chain that the piece is on out to the edge of the board are clear of other pieces. There are many mechanics laid on top of this basic mechanism, such as ice that melts as other pieces escape from the same chain of cells as it, keys that are buried in locked boxes that must be opened with other pieces (also buried in locked boxes), chained pairs of pieces, pieces tied with ropes that must be cut with scissors, timed explosive pieces, etc. A good level is a knot – it looks like a knot, it is solvable, and it is of reasonable size.
Why a workflow can't design a level
When we looked at the numbers generated by the levels designed by the workflow compared to the vast repository of levels hand crafted by the studio for actual game play, the flatness was quantified to three numbers.
Depth of workflow levels vs the shipped corpus
Data table
| Metric | Shipped | Workflow |
|---|---|---|
| Escape waves | 20.5 | 6 |
| Knots (forced convergences) | 4 | 1 |
| Longest piece | 15 | 5 |
What flat looks like
The 6 images of the workflow for a campaign through 18 sample levels of escape depth show short pieces and long weaving pieces which are the depth carriers in the game. In addition to this the images also reveal a very serious bug in the ‘generate a valid board for a given number of players’ stage of the pipeline. All 18 levels were created using only two distinct boards. The pieces on each board were then shuffled to create the various levels.
Six of eighteen workflow samples
Two verdicts, same gates
The workflow's sample
0/182 distinct boards across 18 levels
Median escape depth 6 waves; corpus median 20.5
Features present but not load-bearing
The agent, by the final campaign
50/5012 board families in one run, zero adjacent repeats
Gate medians 26 / 33 / 43 by difficulty tier
Every feature verified load-bearing before save
Notice that every generation workflow is a fixed sequence of generation steps. Irrespective of the number of LLM calls in the generation workflow, a workflow is a fixed sequence of steps. As such, a fixed sequence of generation steps is procedural generation. Each step in the workflow was designed to solve a local problem. The board generator generates a board. The placer placer places pegs on the generated board. The validator validates the solvability of the generated and placed pegs on the board. This means that every one of the quality features that you might like in a level (e.g. deep escape order, interesting knots that force converging solutions, peak of tension late in the level) are global, emergent properties of the entire arrangement of steps in the workflow.
Design is not a line of events. Design is a loop: design, play, find out what doesn’t work well, change that, play again. Design needs to be judged in the middle of that process, not after the whole thing has been generated. To design something is to have tools in the hands of something with taste and following a set of standards. This is an agent.
So we took the project: deconstruct first, prompt later
(The order in which I developed these steps is shown in Figure 05, below. Initially, there was no “Agent”. And, note that the step labeled “Tools + gates” was actually developed in weeks 3–5 of the course!)
The sequence that ended up mattering
Solvability by construction was a two part approach to building these boards, the first part being that every board that was shipped was analyzed through corpus analysis to measure up against every other shipped board, and then the board was run through a full Simulator of the rules of the game in question to test for escapeability in terms of exact escape order. The second part was that all of the dimensions of the game were laid out in a clear and coherent order. In short, all of the Waves and all of the Knots and all of the Traps for a game were laid out in a clear dimensional space prior to beginning to build the boards.
Simulator Test All Shipped Levels - 'solvability' as a sequence of events for each level in the corpus for all shipped levels (through histogram/normal distribution etc.). Means in simulator.
When people talk about deterministic tools, they usually mean that the tool itself is somehow determinate — i.e., guarantees something — and then the workflow somehow fixes up the rest to produce quality. But that’s the problem with the workflow in the first place: it attempts to fix up the sequence of steps to produce quality, whereas what you really want is for the agent to use whatever tools are at its disposal, to produce quality in the first place. The sequence is not determinate.
First contact: three of four rejected
Our first test cases for the levels were the designers and other team members at the studio. They played through a number of the levels to test them out for us, and we got some interesting results. A lot of the levels got into a deadlock where pieces faced off against each other in a standoff. Occasionally a level was impossible to playtest for. After running seventeen of our levels through this process, four of them, or 24%, had passed.
These rules became data to analyze. Every comment of the reviewers resulted in a new rule for the judge script (e.g. “This unlocks in one move — it should be hidden like a key” becomes a rule for features that trigger other features: they have to be in escape-wave 2 or deeper).
- “This unlocks in one move — it should be hidden like a key” → a burial rule: trigger features must sit at escape-wave 2 or deeper.
- “Too many pieces free at once” → a wave-width cap per difficulty.
- “The counter ignores the pieces in front” → A budget floor for every explosive’s counter that is determined by the measured escape order.
- “Feels Normal, tagged Hard” → A knot and trap floor for each tier.
Then, to our surprise, we just compiled the reviewer. The rules were added to the judge script and carefully calibrated to exactly reproduce the human reviewer’s decisions on all 17 levels that had been reviewed so far. Then, in subsequent generations, every single level was judged by this script, before ever being reviewed by a human, and rejected by the save tool if it failed to meet the reviewer’s criteria.
The mistakes were ours too
The workflow for failure in generating these simple levels was clear to retrospect on. It existed in data for the project, and can thus be recounted here too.
The shipped deadlock
Our own failure modes, and what each became
1 · We shipped deadlocks
Two different failures were discovered in the first delivery of levels that could not be completed by players: head-to-head facing pairs, as well as pieces with heads that cross each other’s tails. These failures were not ones that required special care, but rather were failures of construction that were solved in the end by construction: by placing pieces in reverse escape order so that there was always a clear path to escape, and by replaying the full tap sequence of a level through the simulator before allowing it to be saved by the author.
2 · Features as decoration
These elements, including the cuttable rope, key on open padlock, and chained pairs, were included in levels as what seemed to be ‘extra’ features. Although they were cuttable, they added no value to the completion of a level. In fact, in many instances, they only served to confuse players. A load-bearing test per feature would have demonstrated that these elements added nothing to the solve of a level. If removal of a feature resulted in no change to the solution then it was, by definition, decoration. It should have been rejected by review.
3 · One fingerprint isn't originality
Automating the search for duplicate puzzles led to finger printing puzzles (i.e. escape structures) and we found one nice board configuration for puzzles which the generator then created hundreds of times more. All of these created levels were technically distinct but looked all the same. I had misunderstood originality. It is two-axis: shape (silhouette) of the puzzle and the actual puzzle itself.
4 · Trusting the success rate
The success rate of runs (acceptance rate of the pavement) does not correlate with quality (stiffness of the completed level). It is easy to write a run that keeps succeeding while quality is slowly but surely drifting down. In other words: success rate only measures the quality of the enforcement layer, not the quality of the levels that are accepted. Gate depth, drift, and variety need to be measured for each run against the corpus distribution.
The bar moves every time you clear it
As more accepted sets of levels were added to the corpus of accepted levels, the standard for future levels became higher and higher. Levels with multiple floors could be created and were considered “hard floors” for future levels. The bands for opening moves were made tighter and tighter and levels with pressure features were marked as “mandatory” for the top tier of levels. The rule-mining that occurred in 5 rounds of feedback is listed out below:
Five rounds, each ending as code
| Round | What the feedback taught | Became |
|---|---|---|
| 1 | Full boards, originality vs corpus, hole placement | validity gates |
| 2 | Difficulty bands, pressure features, trigger burial | tier floors |
| 3 | Feature ordering, required mechanics, measured flagships | review gate |
| 4 | Opening bands, effective depth, explosive margins | per-tier policy |
| 5 | Gate-aware pressure, separation, pacing | field recipes |
Three eras, side by side
First delivery → encoded rules → final campaign
And the figures climbed with it:
Share of levels passing review
Data table
| Stage | Passing | Of | Share |
|---|---|---|---|
| Studio's shipped corpus | 25 | 60 | 42% |
| The workflow levels | 0 | 18 | 0% |
| Agent — first review round | 4 | 17 | 24% |
| Agent — post rule-mining | 12 | 12 | 100% |
| Agent — final 50-level campaign | 50 | 50 | 100% |
What the levels look like
Above, three examples of levels from the final campaign – one for each of the three difficulty levels (Regular, Hard, and Very Hard) – shown as they were delivered to us (i.e., rendered for web display here). A piece’s color indicates its type. An uppercase letter atop a piece announces a special feature (e.g., I = ice, C = chain). Dark-colored cells represent holes.
One level per difficulty tier
Twelve board families, one campaign
In terms of variety, the studio locked down scheduling of the fifty briefs to lock down 12 board families (above, one silhouette from each family) and 9 different mechanics. So although there are 50 levels, for example, neighboring levels will not have the same board or be of the same type.
Twelve board families, one silhouette each
The final campaign
Instead, for the final campaign all 50 briefs were processed in a single unattended run. Numbered 201-250, the levels ran on a scheduled basis and employed 9 different mechanics and 12 different board families, so that no two adjacent levels were created with the same families. Of the 50 levels five were defined as Hard and five as Very Hard.
Ninety-seven minutes later the fifty levels were done. 45 of the 50 briefs worked on the first try, every subsequent try then arrived at the solution within 4 attempts in every case. The increase in attempts however focused on the right areas:
Attempts needed per level, by difficulty tier
Data table
| Tier | Levels | Avg attempts |
|---|---|---|
| Regular | 40 | 1.20 |
| Hard | 5 | 2.00 |
| Very hard | 5 | 2.20 |
You shipped it. Terms and conditions for the future work of the studio were agreed and represent the healthiest sign that all of this is working at all: one revision pass through all files and thereafter every generation of output of the agent is billed at the same price as output from a designer.
The accidental experiment
Most useful accidental finding for the year:
I experimented with a very modular version of the agent code: 34 fine-grained tools are now 7 clean tools, the floors for the policy are now stored in data not code, and the architecture is better. I ran two 50-level test sets with the same model and same simulator and same enforcement settings as before. Here are the quality results:
Flagship gate score by difficulty tier
Data table
| Tier | Baseline | Rewrite A | Rewrite B |
|---|---|---|---|
| Regular | 26 | 7 | 4 |
| Hard | 33 | 19 | 30 |
| Very hard | 43 | 23 | 38 |
The diff is quite simple here: I had retained one of the two layers of the existing agent and removed the other one.
What the rewrite kept and dropped
What the tools refuse
Per-tier floors as data (piece bands, knots, traps, margins)
The review gate: save refuses a failing level
Solvability by construction
What the prompt aims for
The 128-line prompt shrank to 10; field recipes gone
Flagship targets: declare a gate depth, hold it through review
Theme discipline, palette bands, variety rotation
One thing in particular stands out from this test case: The simple enforcement of floors that were already coded to work worked just fine, while the introduction of the same levels of excellence in terms of instruction to the AI resulted in completely different (and much worse) levels.
We found two more smaller issues in this set of tests, of the class-of-bug type, where instead of failing tests, the AI would silently produce lower quality work.
- Sampling parameters are per-model facts. This is a simple fact about the newest model family that does not get to decline to perform when an API temperature is set (in contrast to the old models, which do decline to perform when an API temperature is set at all and must have it pinned to 0). The basline handled the sampling parameters for each model as a per-model fact; the rewrite has them all as a single fact, hardcoded to the worst value for the hardest brief. Near-duplicate boards on adjacent levels.
- Timeouts are quality parameters too. We noticed in the test cases for the very large models (like DoomSketch big) that actually implement a timeout parameter for the model that can be set at the API level for the newer models (not at framework level for older models) that atimeout had to be set for the largest models (to allow them to run to completion on long design steps), and that had to be set (for the largest models) at the framework level (in the Baseline, this had been raised from the default of 300 seconds or so); in the rewritten prompt, this had been lowered back to the default (so that very long design steps failed to complete in the middle of the design – ie. did not complete at all).
What we'd tell you to steal
- Build the simulator before the agent. Just as the provably correct elevator was developed first and then turned into an agent, the kind of guarantees you want for your levels can only be created by a deterministic system that your model controls, i.e. by simulator code, not model code.
- Measure against the corpus, not against taste. The shipped levels define the distribution; the agent's job is to land on it or explain why not.
- Compile the reviewer. Once you have gotten the feedback from the human — turn it into a rule! That rule then runs on every subsequent level for free.
- Let the bar rise. It is easy to set up an acceptance test that fails in two rounds of development. But then the test is stale. The only good test is one that fails from time to time because it isbeing used. Every review that finds a new failure mode causes the gates to rise and success rate to climb.
- Separate, but keep both enforcement and instruction. Write floors in code. Set high ambitions in prompt. The rewrite experiment showed that, as opposed to floors, ambitions are easy to refactor. So, keep only the layer that is easy to refactor.
All figures were collected from run ledgers and review records, including the number of human reviewers who verified levels before they were passed to automated judges for later stages of review.
