How can a game keep only the good generated levels?

What you're seeing

The live feed is a conveyor. New rooms enter on the right and slide left, each a 16 by 9 grid of tiles: black solid blocks, spikes, ladders, coin rings, square enemies, a start box on the left edge and a red exit flag on the right. Under each room is its stamp. UNSOLVABLE means code found no route from start to exit and never made a call. JUDGING means a request is in flight. ACCEPTED (in red) or REJECTED comes with the room's total and the reason, either its theme score or "under bar". A black REMIX tag marks a near copy of a room that was recently kept.

The eight counters under the feed track the pipeline: rooms generated, pre-filtered, judged and accepted, the acceptance rate, and the live calls per second, tokens per second and dollars per hour. The Taste window holds the four weights, the bar, the theme sentence and the generator rate. Play the survivors is a tiny platformer with the accepted queue under it, best total first. The Jev inspector shows the last judged room: the request's state (the room as ascii, its metrics, and the five recent rooms it was compared with), the probability bars for each rubric level, the theme gauge, and the composite worked out with your current weights.

A few things to try:

• Drag the bar from 0.62 down to 0.50. In a test run that took the queue from 15 rooms to 29. Nothing was re-judged: the answers were already stored on each room, so only the arithmetic changed.
• Push novelty to 1.0. The line under the sliders says the weights now sum to 1.80, so a perfect room totals 1.80 and the bar means less than it did. That's the cost of skipping normalization at the weight level, and the page is honest about it.
• Tap the enemy gauntlet chip. Rooms judged under the old theme leave the queue at once, and the feed starts stamping a different set of rooms.
• Type a theme the mock can't read, such as "a haunted castle". The queue empties and the note under the theme box explains why: none of its words are in the mock's vocabulary, so theme fit sits near 0.5 and nothing clears the 0.6 gate.
• Tap a REMIX room in the feed. Its novelty bars pile onto "a near copy of a recent room", which is the question doing its job.
• Press Play. Arrow keys or A and D to move, Space or W to jump, up and down on ladders, or the touch buttons on a phone. Spikes, enemies and pits reset the room, and the exit flag loads the next room in the queue.

What goes into each judgment call?

Jev is a System One model: it reads state and answers typed questions instead of writing text. The creature brains sim put a paragraph of prose in the state; this one puts a whole level there, drawn as text. Every solvable room becomes one request. Here is the state from a room this demo kept, room 109, trimmed to one of its five recent rooms:

{
  "room_ascii": "################\n....#.....#.....\n................\n..^...o.........\n..#H####........\n.ooH.....o.^o...\n.#####...#####..\nS.^............X\n#####.........##",
  "tile_legend": {
    "#": "solid",
    ".": "empty",
    "^": "spike",
    "H": "ladder",
    "o": "coin",
    "e": "enemy",
    "S": "start",
    "X": "exit"
  },
  "metrics": {
    "solvable": true,
    "path_length": 23,
    "max_gap": 9,
    "spike_count": 3,
    "enemy_count": 0,
    "coin_count": 5,
    "platform_count": 4,
    "vertical_span": 4,
    "empty_ratio": 0.639
  },
  "theme": "underground cave with vertical climbing",
  "recent_accepted": [
    {
      "ascii": "################\n...#....#.#.....\n...............X\n.....^^..e.o####\n...########H....\n....##.....H....\n....######.H....\nS...o..o..eH....\n################",
      "metrics": {
        "path_length": 20,
        "max_gap": 0,
        "spike_count": 2,
        "enemy_count": 2,
        "coin_count": 3,
        "vertical_span": 5,
        "empty_ratio": 0.528
      }
    },
    "...four more kept rooms in the same shape"
  ]
}

The recent_accepted list is what makes novelty answerable at all. The model keeps nothing between calls, so "is this room new" only means something if the last five kept rooms travel with the question. Those five rooms are close to half of every request. Five questions ride along in the same call, four Scores and one Noul:

{
  "fairness": {
    "type": "score",
    "instructions": "How fair is the route from S to X in room_ascii for a player who jumps up to 2 tiles high and across gaps of up to 3 tiles?",
    "criteria": [
      "unwinnable, or needs a jump that is not possible",
      "winnable only with pixel perfect timing",
      "fair, a clear path with real risk",
      "trivial, nothing to do"
    ]
  },
  "readability": {
    "type": "score",
    "criteria": [
      "cluttered, the path is hard to see",
      "mostly clear",
      "the route reads at a glance"
    ],
    "...": "instructions as above"
  },
  "novelty": {
    "type": "score",
    "criteria": [
      "a near copy of a recent room",
      "a variation on recent rooms",
      "a different idea from anything recent"
    ],
    "...": "instructions as above"
  },
  "fun_shape": {
    "type": "score",
    "criteria": [
      "flat and empty",
      "one note",
      "a rhythm of risk and reward",
      "a memorable set piece"
    ],
    "...": "instructions as above"
  },
  "theme_fit": {
    "type": "noul",
    "instructions": "The room matches the theme sentence"
  }
}

Every level is written as a situation the model can look for in the room, not as a degree. "Winnable only with pixel perfect timing" is something you can check against the ascii; "pretty hard" isn't. For room 109 the fairness answer split almost evenly, 0.472 on "pixel perfect" and 0.501 on "fair", so the score came back at 1.50, between the two levels. A Score answer is the probability-weighted mean of the level indices, so a split like that is information, not noise. Each Score also carries a confidence, and the confidence-gated companion tests how far a number like that can be trusted. Readability came back at 0.62 of 2, novelty at 1.87 of 2, fun shape at 2.62 of 3, and theme fit at 0.82. The request was 791 input tokens.

How does code turn five answers into keep or drop?

The model never decides whether a room ships. It returns five judgments, and a few lines of code combine them:

for each candidate:
    room = generate(seed)                               // cheap, local
    if not solvable(room): prefiltered += 1; continue   // code check, no call
    state = { room_ascii, metrics, theme, recent_accepted: last_5_kept }
    a = jev(state, { fairness, readability, novelty, fun_shape, theme_fit })

    norm(q)       = a[q].score / (levels(q) - 1)        // every score onto 0 to 1
    fairness_term = 1 - abs(norm(fairness) - 0.67) * 2  // peaks at "fair"
    total = w_fair * fairness_term + w_read * norm(readability)
          + w_novel * norm(novelty) + w_fun * norm(fun_shape)
    keep  = total >= bar and a.theme_fit.noul >= 0.6

For room 109 with the default weights, that works out as follows. Fairness normalizes to 0.501 and its term is 1 minus twice the distance from 0.67, which is 0.663. Readability normalizes to 0.308, novelty to 0.934 and fun shape to 0.874. The total is 0.35 x 0.663 + 0.15 x 0.308 + 0.20 x 0.934 + 0.30 x 0.874 = 0.727, over the 0.62 bar, and theme fit 0.82 is over 0.6, so the room was kept. It was kept despite a middling readability score, because the fun shape answer leaned toward "a memorable set piece" and the fun weight is twice the readability weight. Move those two sliders and the same stored answers can drop it.

Why four narrow questions beat "is this room good"

Asking a model whether a level is good is the LLM as judge pattern at its bluntest: one rubric, one number, and no way to tell afterward why a room lost. Splitting the judgment into four questions changes three things. Each rubric can be written as concrete situations, which is what makes a score repeatable. The answers come back separately, so the inspector can show that a room lost on novelty and not on fairness, and the designer knows which rubric to fix. And the weights move out of the prompt and into code, where a designer can tune them at runtime without rewriting anything the model reads.

The extra questions are cheap because they share one call. The four rubrics add a few hundred tokens of criteria text to a request that already carries the room and its five neighbors. The answers are structured output by construction: a score, a probability per level, and a confidence, with nothing to parse.

Why normalize first, and why fairness has a sweet spot

Fairness and fun shape have four levels (scores 0 to 3), while readability and novelty have three (0 to 2). Add the raw scores and a four-level question pulls half again as hard as a three-level one before any weight touches it, so a weight of 0.3 stops meaning 30 percent. Dividing each score by its top level puts all four on 0 to 1 first. With the default weights summing to 1, a perfect room totals exactly 1 and the bar reads as a fraction of perfect.

Fairness then gets one more step, because more isn't better at the top of that scale. "Trivial, nothing to do" is as bad a room as "winnable only with pixel perfect timing", just in the other direction. So code measures distance from the "fair" level (2 of 3, which normalizes to 0.67) and turns it into a term that peaks at 1 for fair and falls to about 0.33 at both "pixel perfect" and "trivial". It goes below zero for "unwinnable", which drags the total down hard. The model only reports what kind of room it sees. What the game wants is a design decision, and it lives in code.

Why code rejects unsolvable rooms first

This pipeline is generate and test, the oldest pattern in procedural content generation: make candidates cheaply, then filter. It's also best of N with a twist, since the generator is a random function rather than a model and the scorer is the expensive part. That makes the ordering of tests matter. Whether a player can get from start to exit is a graph question, and a flood fill over standing tiles (jumps up to 2 tiles high, gaps up to 3 tiles wide) answers it in microseconds. In a 20 second run at 8 rooms a second, 64 of 160 rooms failed it. Sending those to a model would pay for a judgment code already has.

The fairness rubric still keeps its "unwinnable" level. The flood fill is coarse about headroom and timing, so the model is a second check on rooms that pass on paper but play badly. Cheap exact checks go first, and the model only judges what code can't.

What does judging hundreds of rooms a minute cost?

At the generator's top speed of 8 rooms a second, the same 20 second run judged 94 rooms and kept 15. The counters read 4.8 calls a second and about 3,800 input tokens a second, which the meter prices at $0.57 an hour. That's about 290 judgments a minute. A request is about 570 tokens when no rooms have been kept yet and about 780 once five recent rooms ride along. At $0.042 per million input tokens with output free, a thousand judged rooms cost about three cents.

Each judgment takes 70 to 500 ms, so one call at a time would fall behind the generator. The dispatcher keeps up to six requests in flight, the way any network client would, and the feed's title bar shows the count. Concurrency has one visible side effect: rooms sent at the same moment are compared against the same five recent rooms, so two similar rooms in flight together can both look new. A production pipeline would either accept that or check novelty again in code before publishing a batch.

The judgments also stay useful after they land. Every room keeps its five answers, so changing a weight or the bar re-ranks everything already judged for free. Only the theme needs new calls, because theme fit is a question about the sentence. The verify-after sim uses the same idea at the other end of a workflow: check every output, because checks are cheap enough to run on all of them.

Frequently asked questions

Key takeaways

• Generation is cheap and judgment is the product. Put exact checks such as solvability in code first, and spend model calls only on candidates that can be finished.
• Ask one question per dimension, with levels written as situations. Separate answers tell you why a room lost, and the weights live in code where a designer can tune them at runtime.
• Normalize every score by its top level before weighting, or questions with more levels quietly outweigh the rest.
• Not every rubric wants a maximum. Fairness peaks at "fair" and falls off toward both "trivial" and "pixel perfect", and that preference belongs to the game, not the model.
• Store the answers. Re-ranking by new weights costs nothing, while context-dependent judgments such as novelty and theme fit need the context in the state and a new call when it changes.