What changes when a game companion acts only when confident?

What you're seeing

You are the white circle and the hound is the black shape. Each of the eight rooms has a right order of operations: pull the lever before the door, light the torch before you open anything in the dark, guard the hero when health is low instead of attacking. The room name, your health, what the hound is carrying, and the room timer sit above the dungeon. The chips and the text box under it send the hound a short command, such as "get the key" or "guard me".

The threshold window holds one number, from 0.20 to 0.95, with a word for its range: reckless, eager, balanced, careful, or asks about everything. Above that confidence the hound acts at once, and a floating label shows the action and its confidence. Below it the hound stops and asks you to pick between its top two actions, with their percentages. Leave the question for 2 seconds and the hound just follows you.

The tally counts the hound's own actions, the questions it asked, and its mistakes. A mistake is an action the room rules say was wrong. Some mistakes also cost something: attacking when it should have guarded lets an enemy hit you, and opening a closet lets enemies out. The calibration chart sorts every decision the hound made on its own into confidence buckets, and each bar's height is how often those decisions were right. The dashed diagonal is where a perfectly calibrated companion would sit. The inspector shows one request going out every 300 ms and the answer in full, with a confidence pill marked by whether it cleared your threshold.

A few things to try:

Drag the threshold down to 0.20 and play two rooms. The hound almost never asks. Watch the mistakes counter.
Push it up to 0.90. Now it asks on nearly every real decision. Count how often option 1 is the right one and how often it is option 2.
Type a command before it gets a chance to ask. An explicit command such as "go for the lever" raises the confidence of the matching action, so a hound that was asking starts acting.
Switch the chart to overconfident mode. The question never appears, every bar piles into the top bucket, and the mistakes keep coming. The gray outlines keep the calibrated bars on screen for comparison.
Tick "include asked decisions". The chart then fills in the buckets below your threshold too, each question scored by its top pick.

One call, three questions, every 300 ms

Jev is a : it reads state and answers typed questions instead of writing text. The pattern runs through this whole series, from the spellcrafting demo to the game director. Here the state is the room: where the hero and the hound are, the hero's health and last command, every visible object and enemy, and the last five events. This is a real state captured from the demo in room 7, the dark ambush room, with the torch still unlit and two enemies asleep:

{
  "room_id": "room_7",
  "hero": {
    "pos": [
      5.5,
      6.5
    ],
    "hp": 65,
    "facing": "east",
    "last_command_text": null,
    "last_command_age_s": null
  },
  "companion": {
    "pos": [
      3.3,
      8
    ],
    "carrying": "nothing",
    "stamina": 1
  },
  "objects": [
    {
      "id": "torch_2",
      "kind": "torch",
      "pos": [
        3.5,
        1.5
      ],
      "state": "unlit",
      "distance_to_hero": 5.4
    },
    {
      "id": "door_7",
      "kind": "door",
      "pos": [
        19.5,
        6.5
      ],
      "state": "locked",
      "distance_to_hero": 14
    }
  ],
  "enemies": [
    {
      "id": "enemy_1",
      "pos": [
        11.5,
        3.5
      ],
      "hp": 30,
      "aggro_target": "none"
    },
    {
      "id": "enemy_2",
      "pos": [
        11.5,
        9.5
      ],
      "hp": 30,
      "aggro_target": "none"
    }
  ],
  "recent_events": [
    "hero entered room 7, ambush in the dark",
    "mistake: hound chose wait, the room rules said follow",
    "hound pulled lever_2",
    "hound brought the potion: hp 65",
    "hound picked up potion_2"
  ]
}

Three questions go with it. action is a choice over eight behaviors, and each description says when that behavior is a mistake, which is where the designer's rules leak into the prompt. target is a choice over the ids in the room plus none, rebuilt on every tick. help_wanted_now is a yes or no check: "The hero needs the companion to act in the next second rather than follow." This is the answer that came back for the state above, trimmed:

{
  "action": {
    "type": "choice",
    "choice": "follow",
    "probabilities": {
      "follow": 0.709,
      "light_torch": 0.18,
      "guard": 0.03,
      "wait": 0.024,
      "...": "four more"
    },
    "confidence": 0.526
  },
  "target": {
    "type": "choice",
    "choice": "none",
    "probabilities": {
      "torch_2": 0.085,
      "door_7": 0.071,
      "enemy_1": 0.087,
      "enemy_2": 0.083,
      "none": 0.674
    },
    "confidence": 0.327
  },
  "help_wanted_now": {
    "type": "noul",
    "noul": 0.491
  }
}

The top action is follow at 0.71, and light_torch is second at 0.18. The confidence is only 0.53: the leader is well ahead, but the rest of the distribution is spread across six other actions. The threshold was 0.80, so the hound asked: "Follow 71 percent" or "Light torch (torch_2) 18 percent". The room rules said light the torch. The question had the right answer in it, as option 2.

Act or escalate

The hound doesn't just run the top answer. It reads the confidence first, and that one comparison decides whether it acts on its own or turns to you. This is the same logic the demo runs:

every 300 ms (skipped while a request is out):
    send({ state: room_state(), model: "jev-latest", questions })

on answer(a):                                          // lands 70 to 500 ms later
    if hound.busy and not (hound.following and a.help_wanted_now.noul >= 0.7):
        return                                         // finish the current job first
    if a.action.confidence >= threshold:
        do(a.action.choice, a.target.choice)           // act alone
    else:
        top2 = two_highest(a.action.probabilities)
        ask_player(top2, timeout = 2 s)                // escalate
        on timeout: do("follow")

This is at game speed, with one twist: the loop is optional and the threshold decides when it closes. Two details keep it playable. An answer only takes effect when the hound is free, because otherwise it would change its mind mid-walk every 300 ms. And help_wanted_now is the interrupt: at 0.7 or more, a hound that is only following or waiting drops that and takes the new answer at once.

The fallback matters as much as the threshold. A question nobody answers turns into follow, the one action that is never harmful. A companion that guesses when ignored is back to acting on low confidence, just later.

Why a flat distribution is information

A model that returned only its top pick would hide the coin flip, and the game would find out by opening the wrong door. The spread is the useful part. When two actions both look reasonable, the probabilities say so, the confidence drops, and the game can turn that into a two-button question that usually contains the right answer. For the player, picking between two good options is a lot cheaper than cleaning up after a closet full of enemies.

In the demo, confidence is 1 minus the normalized entropy of the action distribution, so it measures how peaked the whole distribution is, not just the size of the winner. That is why the captured answer above has a top probability of 0.71 and a confidence of 0.53. TypeSafe describes its confidence the same way, high for one peak and low for a flat spread, though its exact formula isn't published.

Calibration is what makes the threshold mean anything

A threshold of 0.7 is a promise: the hound acts alone only when it expects to be right at least 70 percent of the time. That promise holds only if the reported confidence matches the observed hit rate, which is what means. A calibrated 0.7 is right about seven times in ten, and the chart checks that live.

To test it, a bot played the demo at 8x speed, walking the hero to the exit or the nearest enemy and otherwise waiting. Over five full runs at a threshold of 0.20 (so nearly everything was autonomous), the hound made 284 decisions on its own. Grouped by reported confidence, accuracy against mean confidence came out like this:

0.2 to 0.3: right 20 percent of the time at a mean confidence of 0.25 (15 decisions)
0.3 to 0.4: 42 percent at 0.36 (12 decisions)
0.4 to 0.5: 46 percent at 0.47 (28 decisions)
0.5 to 0.6: 61 percent at 0.55 (66 decisions)
0.6 to 0.7: 54 percent at 0.65 (48 decisions)
0.7 to 0.8: 69 percent at 0.75 (64 decisions)
0.8 to 0.9: 84 percent at 0.84 (44 decisions)
0.9 to 1.0: 100 percent at 0.91 (7 decisions)

Most buckets sit within a few points of the line. The 0.6 bucket is the exception: it ran about 0.1 under the diagonal in this run and again in a second run at a threshold of 0.60 (54 percent at 0.65, 28 decisions). With 28 to 48 decisions per bucket that is within what sampling noise can produce, but it showed up twice, so treat it as a soft spot rather than proof of perfect calibration. That is the honest reading of any calibration chart: buckets with a few dozen decisions wobble, and you need many more before a gap means something.

What overconfidence costs

Overconfident mode keeps the same mock and the same distributions, and changes one number. Reported confidence becomes the larger of the real value and 0.92. Nothing else differs, and the hound picks the same actions it would have picked anyway. But every answer now clears every threshold, so the question never appears.

Over the same stretch of game time, the calibrated hound at a threshold of 0.60 acted alone 87 times, asked 63 times, and made 24 mistakes. The overconfident hound acted alone 183 times and made 75 mistakes. On the chart, all 183 decisions land in the 0.9 bucket at 59 percent accuracy against a reported 0.92: a bar far below the diagonal. A model that is often right but never says when it's unsure can't be given autonomy, because no threshold separates its good calls from its bad ones.

The threshold is also an accessibility setting

The same number that trades speed for accuracy sets how much a player has to manage. At 0.3 the hound is eager, rarely interrupts, and sometimes pulls the wrong thing. At 0.9 it checks in on almost every real decision. Some players want exactly that: someone who wants company but not surprises, or who needs time to read and decide, can move the slider up and get a companion that never acts without asking. A player who finds the questions tedious can move it down.

That setting only works because the confidence is honest. On the overconfident hound the slider does nothing at all, because every answer already reports 0.92 or more. A difficulty or assist option built on model confidence is only as good as the model's calibration, which makes calibration a player-facing feature rather than a research metric.

What it costs at 300 ms

The captured request above is 502 input tokens, and calls average about 450, most of it the eight action descriptions. Output is free. The demo skips a tick while a request is still out, so the measured rate is 2 to 2.3 calls a second rather than the 3.3 a strict 300 ms clock would give. At $0.042 per million input tokens that works out to about $0.14 to $0.16 per hour of play. The questions the hound asks cost nothing extra: they come from an answer the game already paid for.

What is simulated

Nothing in the demo calls Jev, which is in early access. Every answer comes from a local mock, and the mock is calibrated because it cheats: it reads the room rules table, so it knows the correct action. For each call it draws a target confidence from a range set by the situation (an explicit command raises it, an enemy next to the hero raises attack or guard, two valid objects or a health level near the attack or guard cutoff lower it). With that probability the peak of the distribution goes on the correct action, otherwise on a plausible wrong one, and the peak height is solved so that the reported confidence comes out at the drawn value. A real model has no rules table to read, which is why its calibration has to be measured on logged decisions rather than assumed. Correctness is scored on the action only, not the target. Commands are read with a small keyword table. Confidence is 1 minus normalized entropy, an approximation of TypeSafe's unpublished formula. Latency is drawn uniformly from 70 to 500 ms, and token counts are the JSON length divided by 4. The request and response shapes, model names, latency range, and price come from the TypeSafe docs.

Frequently asked questions

What does calibrated confidence mean in AI?

A model is calibrated when its stated confidence matches how often it is right. Of all the answers it gives at 0.7, about 70 percent should turn out correct. You check it by logging decisions, grouping them by reported confidence, and comparing each group's accuracy with its confidence, which is what the chart in this demo does live. A model can be accurate and still badly calibrated, as overconfident mode shows.

How do companion AI characters work in games?

Most are behavior trees or utility systems: hand-written rules that score a fixed set of actions against the game state. The pattern here keeps the fixed action list and the hand-written routines, but asks a model to rank the actions from the room state, then uses the model's confidence to decide whether to act or ask. The creature brains demo shows the same idea with personality text in the state instead of a threshold.

What is human in the loop AI?

It is a design where an AI proposes and a person approves, edits, or rejects before anything happens. The usual version puts a human on every decision. A confidence threshold makes it selective: the system acts alone on clear calls and escalates the uncertain ones, so people review the cases where their judgment adds the most.

Why ask between the top two actions instead of all eight?

Because two options can be read and picked in under two seconds during play, and the low-confidence cases are usually a split between two plausible actions anyway. The captured example in this post had the right answer as option 2. A game with more time per decision could show three. The gate guard demo uses the same closed-choice idea for dialogue lines.

What should a game do if the model is not calibrated?

Measure first. Log the model's reported confidence and whether each decision turned out right, then plot accuracy by bucket. If confidence runs high, raise the threshold until the autonomous decisions are right often enough, or remap reported values to observed accuracy before comparing them with the threshold. What it should not do is trust a threshold it has never checked, because the setting quietly means something different than it says.

Key takeaways

Confidence turns a model's answer into a routing decision: act above the threshold, ask below it, and fall back to the one action that is never harmful when nobody answers.
A flat distribution is a feature. It marks exactly the moments where a two-button question is cheaper than a mistake, and the question usually contains the right answer.
A threshold is only as meaningful as the model's calibration. An overconfident model makes the slider useless, because every answer clears it and no setting separates good calls from bad ones.
Calibration has to be measured, not assumed. Log decisions, bucket them by confidence, and expect buckets with a few dozen decisions to wobble before you trust a gap.
The same threshold is a player setting. Moving it up gives a companion that asks before it acts, which some players want for comfort or accessibility.