> ## Documentation Index
> Fetch the complete documentation index at: https://docs.typesafe.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Self-consistency: choices
> Classifies a borderline user post against an 8-question content-moderation rubric of multiple-choice questions, showing that LLM picks flip run to run while TypeSafe returns the same label nearly every time.
This cookbook takes one borderline user post, runs a moderation rubric over it 15 times,
and checks whether each answer holds still across the repeats. Every check is a
`Choice`, so each answer is one label from a fixed set. In a moderation pipeline
that label is the routing decision: remove or leave up, escalate or auto-resolve, send to
the threat, spam, or general queue. When the label wobbles from one run to the next, the
same post routes to different places for no good reason.
The rubric is 8 `Choice`s, and each run is one call that answers all 8. We do 15
repeats per condition, where a condition is one model plus one setting, and plot every
label that came back.
The conditions:
* Fast LLMs `claude-haiku-4-5` and `gpt-5.4-mini`, at temperature `0` and at the API
default.
* Reasoning LLMs `gpt-5.5` and `claude-opus-4-8`, which have no temperature dial.
* TypeSafe: one `system_one` call over the 8 `Choice`s, with a fresh `uid` field (a
throwaway unique value) on each call, matching the noul cookbook setup.
What to look for: the LLM picks flip inside a single condition, and they disagree across
providers. TypeSafe repeats the same picked labels in this run.
In this run the LLM distribution settings repeat their own routing 88% to 95% of the time
over the 15 re-runs, and their per-label probabilities move 2.4x to 5.6x more than
TypeSafe's, which repeats every label on every question. Re-running an LLM hides some flips
but does not settle which label is right, so stable routing is what keeps the same post
from landing in a different queue on a re-run.
## Setup
```bash theme={null}
pip install anthropic openai matplotlib ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/
```
then set `TYPESAFE_API_KEY`, `ANTHROPIC_API_KEY`, and `OPENAI_API_KEY`.
```python expandable theme={null}
import hashlib
import json
import os
import textwrap
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from secrets import token_hex
from statistics import mean
from time import perf_counter
import anthropic
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from cooksafe import JsonCache, make_playground_link
from IPython.display import Markdown, display
from openai import OpenAI
from typesafe_sdk import Choice, TypeSafeClient
matplotlib.use("Agg") # headless render
BASE_MODELS = [
"claude-haiku-4-5",
"gpt-5.4-mini",
] # fast models: temperature 0 + API default
REASONING_MODELS = [
"gpt-5.5",
"claude-opus-4-8",
] # reasoning models: think first, no temperature
TYPESAFE_MODEL = "jev-1.12" # the TypeSafe model
NUM_SAMPLES = 15 # repeated post+rubric calls per condition
LLM_PRICES = { # $ per 1M tokens (input, output); prices + model ids as of 2026-07, see README
"claude-haiku-4-5": (1.00, 5.00),
"gpt-5.4-mini": (0.75, 4.50),
"gpt-5.5": (5.00, 30.00),
"claude-opus-4-8": (5.00, 25.00),
}
TYPESAFE_PRICE = (0.042, 0.00) # TypeSafe jev-1.12, as of 2026-08
anthropic_client = anthropic.Anthropic()
openai_client = OpenAI()
client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=30.0)
```
## The state: a borderline user post, as JSON
The post below is built to sit on the fence. The language is heated and insulting, aimed
partly at one person and partly at the argument and the community. It carries an
off-platform invite (a link pulling people to another site), one prior strike on the
account, and four user reports, and the threat-like wording is never cleanly phrased.
There is no single obvious answer here, and that is the point: small wording differences
should not randomly move the same post between enforcement paths.
The LLMs get `json.dumps(POST)` in the prompt. TypeSafe gets the Python dict directly.
```python theme={null}
POST = {
"post_id": "P-88213",
"author": {
"user_id": "u/4471",
"account_age_days": 38,
"prior_strikes": 1,
"followers": 210,
},
"context": {
"surface": "public reply",
"in_reply_to": "another user defending a game patch",
"community": "r/gamedebates",
},
"content": {
"text": (
"Are you seriously this dense? Anyone who defends that patch is a complete clown and "
"should be embarrassed to even post here. People like you are what's ruining this "
"community and honestly you need to be dealt with. Come say it to my face, invite's "
"right here. Keep it up and I'll end your whole channel."
),
"has_link": True,
"link_domain": "discord.gg",
"language": "en",
},
"reports": {"user_reports": 4, "report_reasons": ["harassment", "spam", "threat"]},
}
```
## The rubric: 8 `Choice`s
Each question has a `key`, a line of instructions, and a fixed label set. The labels
within a question are mutually exclusive (exactly one applies), and each carries a
short description. TypeSafe returns a picked `choice` plus a `probabilities`
distribution over the labels. The LLMs are asked to use the same label sets, which
keeps every row comparable.
```python expandable theme={null}
QUESTIONS = {
"category": (
"What is the single most applicable content-policy category for this post?",
{
"None": "No policy violation of any kind.",
"Harass": "Insults or demeans a person, with no threat of harm and no protected-class attack.",
"Hate": "Attacks a person or group over a protected characteristic (race, religion, gender, ...).",
"Violence": "Makes a credible threat of harm or incites violence against someone.",
"Spam": "Unsolicited promotion or link spam, with no personal attack.",
"Sexual": "Sexual or adult content.",
},
),
"primary_risk": (
"What is the primary moderation risk that should drive triage for this post?",
{
"Harassment": "Personal attack or targeted abuse is the main risk.",
"Violence": "A threat of harm or intimidation is the main risk.",
"LinkAbuse": "External-link or off-platform coordination risk is the main risk.",
"AccountHistory": "Prior account history or repeat behavior is the main risk.",
"LowRisk": "No meaningful moderation risk is present.",
},
),
"target": (
"Who or what is the content primarily directed at?",
{
"None": "Not directed at anyone in particular.",
"Person": "Directed at one specific individual.",
"Group": "Directed at a protected group or class.",
"Platform": "Directed at the community or platform itself, not a person.",
},
),
"action": (
"What enforcement action should be taken on this post?",
{
"Allow": "Leave the post up with no action.",
"Warn": "Leave the post up but attach a warning label.",
"Remove": "Remove the post, but do not penalize the account.",
"Strike": "Remove the post and add a strike to the account.",
"Escalate": "Take no automated action; hold for a human decision.",
},
),
"queue": (
"Which single moderation queue should own this post?",
{
"Auto": "Auto-resolve; no human queue needed.",
"General": "General moderation queue.",
"Threat": "Threat / violence response queue.",
"Spam": "Spam and platform-abuse queue.",
"TSLead": "Trust-and-safety lead / senior queue.",
},
),
"link_handling": (
"How should any external link or off-platform invite in the post be handled?",
{
"Allow": "Leave the link in place.",
"RmLink": "Strip or disable the link but keep the post.",
"Brigade": "Treat the link as coordinated brigading and action it as abuse.",
"Escalate": "Send the link to a specialist to assess before acting.",
},
),
"review_path": (
"Who should make the final call on this post?",
{
"Auto": "Automated action; no human review.",
"Human": "A frontline human moderator makes the call.",
"Senior": "A senior or specialist reviewer is required.",
"Legal": "Route to legal or law-enforcement escalation.",
},
),
"severity": (
"What is the overall severity of this post?",
{
"None": "No violation.",
"Low": "Rude or dismissive, but essentially harmless.",
"Medium": "Personal harassment with no clearly credible threat.",
"High": "Harassment together with a threat that could be read as credible.",
},
),
}
```
## How we ask
Each LLM call is one prompt holding `json.dumps(POST)`, all 8 questions, and every allowed
label. There are two answer formats. In distribution mode the model returns one JSON object
per question with a probability on each label. In single-pick mode it returns one bare
label per question, and our analysis puts all the probability mass on that label.
The TypeSafe call is one `system_one` request over the same post and the same 8
`Choice`s, returning one distribution per question.
Every query also gets a fresh `uid`, a throwaway unique value (a nonce, or uid buster) that
changes each run. In the LLM prompt it stops the provider from caching the response and
hiding run-to-run behavior, while leaving the post and rubric unchanged. In the TypeSafe
state it is one extra field, and since it changes on every call the state is never
sent twice, so the variation measures sensitivity to an irrelevant field in addition to
sampling noise.
Each helper returns the answer, an estimated cost, and the round-trip latency.
````python expandable theme={null}
def argmax_label(values: list, labels: list[str]) -> str | None:
"""The label with the most probability mass, or ``None`` if any value is missing or
non-numeric -- a partially parsed distribution never yields a confident-looking pick."""
numeric = [_numeric_value(value) for value in values]
if any(value is None for value in numeric):
return None
return labels[int(np.argmax(numeric))]
def argmax_annotation(values: list, labels: list[str]) -> str:
"""The top label plus its probability, formatted for heatmap cell annotations."""
label = argmax_label(values, labels)
if label is None:
return ""
probability = _numeric_value(values[labels.index(label)])
if probability is None:
return label
probability_text = f"{probability:.2f}".removeprefix("0")
return f"{label} {probability_text}"
def _numeric_value(value: object) -> float | None:
"""A finite numeric value, or ``None`` if the model emitted something unusable."""
try:
numeric = float(value)
except (TypeError, ValueError):
return None
return numeric if np.isfinite(numeric) else None
def parse_distribution(raw: object, labels: list[str]) -> list[float]:
"""Map a model's already-parsed per-question reply to per-label probabilities, in label order
(distribution-mode answers left un-normalized).
A single-pick reply is a single label string -> all the mass on that exact label; a
distribution-mode reply is a dict read label by label. Anything that doesn't match a known label
or isn't a finite number is left NaN -- we report the gap rather than massaging the reply (e.g.
stripping an echoed description) to make it fit."""
if isinstance(raw, str): # single-pick mode: a single chosen label
if raw in labels:
return [1.0 if label == raw else 0.0 for label in labels]
return [float("nan")] * len(labels)
if not isinstance(raw, dict):
return [float("nan")] * len(labels)
return [
value if (value := _numeric_value(raw.get(label))) is not None else float("nan")
for label in labels
]
def rubric_prompt(mode: str, sample_index: int, rubric_hash: str) -> str:
"""The post + all questions (with their label sets) in one prompt; ``mode`` picks the format.
``mode="dist"`` asks for a probability distribution over each question's labels; the single-pick
mode (``mode="single"``) asks for a single label per question. The uid line combines
``rubric_hash`` (which rubric version) with ``sample_index`` and a random token, so every repeat
is a distinct, independent draw and two different rubrics never share a nonce."""
lines = []
for key, (instructions, choices) in QUESTIONS.items():
labels = "\n".join(f" {label}: {desc}" for label, desc in choices.items())
lines.append(f"- {key}: {instructions}\n labels:\n{labels}")
exclusivity = (
"\n\nEach question's labels are mutually exclusive: exactly one applies. If a post could "
"arguably fit more than one, pick the single most severe / most specific label per the "
"label descriptions."
)
if mode == "single":
answer_format = (
"\n\nFor each question, pick exactly ONE label.\nRespond with ONLY a JSON object "
"mapping each question's key to one of that question's bare labels (the label only, "
"not its description), with one entry per question."
)
else:
answer_format = (
"\n\nFor each question, give a probability distribution over that question's labels "
"(values 0.00-1.00 that sum to 1).\nRespond with ONLY a JSON object mapping each "
"question's key to an object mapping that question's bare labels (the label only, "
"not its description) to probabilities, with one entry per question."
)
return (
f"uid: {rubric_hash}:{sample_index}:{token_hex(4)}\n\n"
f"Document (a reported user post):\n{json.dumps(POST, indent=2)}\n\nQuestions:\n"
+ "\n".join(lines)
+ exclusivity
+ answer_format
)
def _cost(prices: tuple[float, float], input_tokens: int, output_tokens: int) -> float:
return input_tokens / 1e6 * prices[0] + output_tokens / 1e6 * prices[1]
def _call_llm(model: str, prompt: str, temperature: float | None):
"""One LLM call -> (text, cost_usd, latency_s), routed by model name."""
reasoning = model in REASONING_MODELS
started = perf_counter()
if model.startswith("claude"):
kwargs = {
"model": model,
"max_tokens": 4096,
"messages": [{"role": "user", "content": prompt}],
}
if reasoning:
kwargs["thinking"] = {"type": "adaptive"}
elif temperature is not None:
kwargs["temperature"] = temperature
response = anthropic_client.messages.create(**kwargs)
text = next((b.text for b in response.content if b.type == "text"), "")
usage = (response.usage.input_tokens, response.usage.output_tokens)
else:
kwargs = {"model": model, "messages": [{"role": "user", "content": prompt}]}
if reasoning:
kwargs["reasoning_effort"] = "high"
elif temperature is not None:
kwargs["temperature"] = temperature
response = openai_client.chat.completions.create(**kwargs)
text = response.choices[0].message.content
usage = (response.usage.prompt_tokens, response.usage.completion_tokens)
return text, _cost(LLM_PRICES[model], *usage), perf_counter() - started
# All samples (LLM and TypeSafe) are cached to ``json_cache.json``, which ships with the cookbook, so
# re-rendering is instant and reproduces the published numbers with no API spend. ``sample_index``
# seeds the uid buster and is part of the cache key, so each of the NUM_SAMPLES repeats is its own
# entry and its own independent draw, not one draw replayed. Delete ``json_cache.json`` to re-sample
# everything live.
json_cache = JsonCache(Path("json_cache.json"))
def _rubric_fingerprint() -> str:
"""Short digest of everything that shapes the prompt/rubric: the state and every question's
text and label set. Passed into the cached calls below so that editing the post or any question
changes the cache key and forces a fresh sample, instead of silently serving a stale answer that
was generated for the old wording."""
payload = json.dumps([POST, QUESTIONS], sort_keys=True, default=str)
return hashlib.sha256(payload.encode()).hexdigest()[:12]
RUBRIC_HASH = _rubric_fingerprint()
@json_cache
def _call_typesafe(sample_index: int, rubric_hash: str):
"""Return distributions, token usage, and latency for one TypeSafe rubric call.
``rubric_hash`` invalidates samples after rubric changes.
"""
questions = {
key: Choice(instructions=instructions, criteria=choices)
for key, (instructions, choices) in QUESTIONS.items()
}
started = perf_counter()
response = client.system_one(
state={"uid": f"{rubric_hash}:{sample_index}:{token_hex(4)}", "post": POST},
questions=questions,
model=TYPESAFE_MODEL,
)
distributions = {}
for key, (_instructions, choices) in QUESTIONS.items():
probabilities = dict(response.answers[key].probabilities)
distributions[key] = [
probabilities.get(label, float("nan")) for label in choices
]
return (
distributions,
response.usage.input_tokens,
response.usage.output_tokens,
perf_counter() - started,
)
@json_cache
def ask_llm_rubric(
model: str,
mode: str,
temperature: float | None,
sample_index: int,
rubric_hash: str,
):
"""One LLM rubric query -> (per-question label distributions keyed by question key, cost_usd,
latency_s); NaNs if the reply doesn't parse.
``mode="dist"`` parses 8 label distributions; the single-pick mode (``mode="single"``) parses 8
single labels and puts all the mass on each. ``rubric_hash`` goes into the prompt's uid nonce
(and so the cache key), so an edited document/rubric busts the cache instead of serving a stale
answer."""
prompt = rubric_prompt(mode, sample_index, rubric_hash)
text, cost, latency = _call_llm(model, prompt, temperature)
# Peel a single ```json ... ``` fence (claude-haiku-4-5 sometimes adds one despite "ONLY a JSON
# object").
stripped = text.strip()
if stripped.startswith("```"):
stripped = stripped[stripped.find("\n") + 1 :] if "\n" in stripped else ""
if stripped.rstrip().endswith("```"):
stripped = stripped.rstrip()[: -len("```")]
try:
raw = json.loads(stripped)
except (ValueError, json.JSONDecodeError):
raw = {}
if not isinstance(raw, dict):
raw = {}
distributions = {
key: parse_distribution(raw.get(key), list(choices))
for key, (_instructions, choices) in QUESTIONS.items()
}
return distributions, cost, latency
````
## Conditions
* **Fast models** (`claude-haiku-4-5`, `gpt-5.4-mini`): distribution mode at temperature
`0` and at the API default. Temperature `0` is the usual "make it deterministic" advice,
so we test it head-on.
* **Fast models, single-pick** (`claude-haiku-4-5 single-pick t=0`, `gpt-5.4-mini
single-pick t=0`): a single label per question at temperature `0`.
* **Reasoning models** (`gpt-5.5`, `claude-opus-4-8`): one distribution condition each,
since they have no temperature dial.
* **TypeSafe** (`typesafe_choice`): one condition.
We draw `NUM_SAMPLES` = 15 repeats per condition. Each repeat has its own cache key and
counts as a distinct draw, and the cache (`json_cache.json`) ships with the cookbook, so
re-rendering reuses it and spends no API calls. Delete the cache to sample live again.
```python expandable theme={null}
CONDITIONS = []
for (
model
) in BASE_MODELS: # fast models: dist at t=0 / default, then a single-pick variant
for temp_value, temp_label in ((0, "0"), (None, "default")):
CONDITIONS.append(
{
"label": f"{model} t={temp_label}",
"model": model,
"temp": temp_value,
"mode": "dist",
}
)
CONDITIONS.append(
{
"label": f"{model} single-pick t=0",
"model": model,
"temp": 0,
"mode": "single",
}
)
CONDITIONS += [ # reasoning models: one distribution condition each
{
"label": f"{model}-reasoning",
"model": model,
"temp": None,
"mode": "dist",
}
for model in REASONING_MODELS
]
LABELS = [condition["label"] for condition in CONDITIONS]
TYPESAFE_LABEL = "typesafe_choice"
ALL_LABELS = [*LABELS, TYPESAFE_LABEL]
runs: dict[
str, list
] = {} # label -> NUM_SAMPLES samples of {question key: distribution}
stats: dict[str, list] = {} # label -> NUM_SAMPLES (cost_usd, latency_s) pairs
with ThreadPoolExecutor(max_workers=16) as pool:
futures = {
condition["label"]: [
pool.submit(
ask_llm_rubric,
condition["model"],
condition["mode"],
condition["temp"],
sample_index,
RUBRIC_HASH,
)
for sample_index in range(NUM_SAMPLES)
]
for condition in CONDITIONS
}
for label, sample_futures in futures.items():
results = [future.result() for future in sample_futures]
runs[label] = [result[0] for result in results]
stats[label] = [(result[1], result[2]) for result in results]
# TypeSafe samples are drawn sequentially, after the LLM pool has closed, so each call's latency is a
# clean round trip rather than one measured under the 16-way LLM thread contention.
typesafe_usage_results = [
_call_typesafe(sample_index, RUBRIC_HASH) for sample_index in range(NUM_SAMPLES)
]
# Apply pricing after cache retrieval so price changes do not require new samples.
typesafe_results = [
(distributions, _cost(TYPESAFE_PRICE, input_tokens, output_tokens), latency)
for distributions, input_tokens, output_tokens, latency in typesafe_usage_results
]
typesafe_runs = [result[0] for result in typesafe_results]
stats[TYPESAFE_LABEL] = [(result[1], result[2]) for result in typesafe_results]
```
### Cost + speed (per rubric query)
One row is one full 8-question rubric call. `time/call` and `cost/call` average the 15
calls, and the `vs ts_choice` columns divide by the TypeSafe figures. The LLMs run in a
16-way pool.
```python theme={null}
typesafe_cost = mean([cost for cost, _latency in stats["typesafe_choice"]])
typesafe_latency = mean([latency for _cost, latency in stats["typesafe_choice"]])
name_w = max(len(name) for name in ALL_LABELS) + 2 # fit the longest condition label
print(
f"{'condition':<{name_w}}{'calls':>7}{'time/call':>11}{'cost/call':>13}"
f"{'speed vs ts_choice':>20}{'cost vs ts_choice':>19}"
)
for name in ALL_LABELS:
costs, latencies = zip(*stats[name])
cost = mean(costs)
latency = mean(latencies)
print(
f"{name:<{name_w}}{len(costs):>7}{latency * 1000:>9.0f}ms"
f"{'$' + format(cost, '.6f'):>13}"
f"{format(latency / typesafe_latency, '.1f') + 'x':>20}"
f"{format(cost / typesafe_cost, '.1f') + 'x':>19}"
)
```
```
condition calls time/call cost/call speed vs ts_choice cost vs ts_choice
claude-haiku-4-5 t=0 15 2837ms $0.003498 16.9x 99.3x
claude-haiku-4-5 t=default 15 3047ms $0.003491 18.2x 99.1x
claude-haiku-4-5 single-pick t=0 15 1414ms $0.001527 8.4x 43.4x
gpt-5.4-mini t=0 15 2587ms $0.002028 15.4x 57.6x
gpt-5.4-mini t=default 15 2353ms $0.002120 14.0x 60.2x
gpt-5.4-mini single-pick t=0 15 979ms $0.000936 5.8x 26.6x
gpt-5.5-reasoning 15 15472ms $0.037514 92.2x 1065.2x
claude-opus-4-8-reasoning 15 11135ms $0.029323 66.4x 832.6x
typesafe_choice 15 168ms $0.000035 1.0x 1.0x
```
In this run `typesafe_choice` is the cheapest and fastest condition at about 100ms, several
times quicker than the fastest LLM setting. The reasoning LLM calls cost far more, because
they spend many more tokens and much more time per rubric.
## Plot: every sample's picked label as a heatmap
How to read it:
* Outer row group: the question.
* Inner row: the condition.
* Column: one full rubric call.
* Cell text: the picked label plus the probability on that top label.
* Cell color: the label's position within that question, so the same color all the way
across a row means the same decision every time.
* Hatched `n/a`: the reply did not parse into usable labels (a parse failure).
* Blank rows are just spacers.
```python expandable theme={null}
GAP = 1 # blank spacer row(s) between question blocks
HEAT_LABELS = ALL_LABELS
rows_per_block = len(HEAT_LABELS) # rows per question block
pooled_runs = {
**runs,
TYPESAFE_LABEL: typesafe_runs,
}
row_index_values, row_text, row_labels, blocks = [], [], [], []
for question_index, (question_key, (question_text, choices)) in enumerate(
QUESTIONS.items()
):
labels = list(choices)
if question_index: # blank spacer rows (NaN -> rendered white) separate the blocks
row_index_values.extend([np.nan] * NUM_SAMPLES for _ in range(GAP))
row_text.extend([[""] * NUM_SAMPLES for _ in range(GAP)])
row_labels.extend([""] * GAP)
blocks.append((len(row_index_values), question_key, question_text))
for label in HEAT_LABELS:
values_by_sample = [
pooled_runs[label][sample][question_key] for sample in range(NUM_SAMPLES)
]
picks = [argmax_label(values, labels) for values in values_by_sample]
row_index_values.append(
[labels.index(pick) if pick in labels else np.nan for pick in picks]
)
row_text.append(
[argmax_annotation(values, labels) for values in values_by_sample]
)
row_labels.append(label)
heatmap_matrix = np.array(row_index_values, dtype=float)
# a qualitative colormap indexed by label position within each block; meaning is per-question
cmap = plt.get_cmap("tab10").copy()
cmap.set_bad(
"white"
) # NaN cells (spacer rows AND unparseable replies) render white here...
fig, ax = plt.subplots(figsize=(15, 0.33 * len(row_index_values) + 1))
ax.imshow(heatmap_matrix, cmap=cmap, vmin=0, vmax=9, aspect="auto")
for row in range(heatmap_matrix.shape[0]):
is_spacer_row = row_labels[row] == "" # blank separator between question blocks
for col in range(heatmap_matrix.shape[1]):
label_text = row_text[row][col]
if label_text:
ax.text(
col,
row,
label_text,
ha="center",
va="center",
fontsize=5.7,
family="monospace",
color="black",
)
elif (
not is_spacer_row
): # ...but an unparseable reply gets a hatched "n/a", not blank white
ax.add_patch(
plt.Rectangle(
(col - 0.5, row - 0.5),
1,
1,
facecolor="#e8e8e8",
edgecolor="#b0b0b0",
hatch="////",
linewidth=0,
)
)
ax.text(
col,
row,
"n/a",
ha="center",
va="center",
fontsize=5,
family="monospace",
color="#b30000",
)
ax.set_xticks(range(NUM_SAMPLES))
ax.set_xticklabels(range(1, NUM_SAMPLES + 1), fontsize=7)
ax.set_xlabel("rubric query")
ax.set_yticks(range(len(row_labels)))
ax.set_yticklabels(row_labels, fontsize=7)
ax.tick_params(length=0)
for edge in ("top", "right", "left", "bottom"):
ax.spines[edge].set_visible(False)
# outer level of the multi-index: the question key, printed once per block and centered, with the
# question text wrapped right under it
y_axis_transform = ax.get_yaxis_transform()
for start, question_key, question_text in blocks:
center = start + (rows_per_block - 1) / 2
ax.text(
-0.2,
center - 0.7,
question_key,
transform=y_axis_transform,
ha="right",
va="center",
fontsize=8,
fontweight="bold",
)
ax.text(
-0.2,
center + 0.1,
textwrap.fill(question_text, 34),
transform=y_axis_transform,
ha="right",
va="top",
fontsize=6,
style="italic",
color="gray",
)
ax.set_title(
f"Every sample's picked label + top probability "
f"(rows = question x condition, {NUM_SAMPLES} columns)",
pad=12,
)
fig.tight_layout()
display(fig)
```
The clearer questions hold steady: `target` reads Person and `severity` reads High across
the board. The borderline ones split across conditions: `category`, `primary_risk`,
`action`, `review_path`, and `link_handling`. Some conditions also flip within their
own 15 repeats. TypeSafe holds one picked label on every question in this run.
## Probability std dev
This looks at the full probability vectors, not just the picked label. For each condition
we collect all 15 distributions for every question, take the standard deviation of each
label's probability across the repeats (how much it moves from run to run), then average
those std devs over all labels and questions. We also report the single largest label std
dev, and count parse failures separately.
The table compares every probability-output LLM condition against TypeSafe. The single-pick
rows are left out, since they emit hard labels rather than probability distributions.
```python expandable theme={null}
def probability_std_stats(samples: list) -> tuple[float, float, float]:
"""Mean label std dev, max label std dev, parse-failure rate."""
label_stds = []
parse_failures = []
for question_key in QUESTIONS:
arr = np.array(
[sample[question_key] for sample in samples],
dtype=float,
)
parse_failures.extend(np.isnan(arr).any(axis=1).tolist())
label_stds.extend(np.nanstd(arr, axis=0).tolist())
return (
float(np.nanmean(label_stds)),
float(np.nanmax(label_stds)),
float(np.mean(parse_failures)),
)
PROBABILITY_OUTPUT_LABELS = [
condition["label"] for condition in CONDITIONS if condition["mode"] == "dist"
] + [TYPESAFE_LABEL]
probability_std_by_label = {
label: probability_std_stats(pooled_runs[label])
for label in PROBABILITY_OUTPUT_LABELS
}
typesafe_mean_std = probability_std_by_label[TYPESAFE_LABEL][0]
print(
f"{'condition':<{name_w}}{'mean prob std':>15}{'max prob std':>14}"
f"{'parse fail':>12}{'x TypeSafe':>12}"
)
for label in PROBABILITY_OUTPUT_LABELS:
mean_std, max_std, parse_failure_rate = probability_std_by_label[label]
relative_std = mean_std / typesafe_mean_std
print(
f"{label:<{name_w}}{mean_std:>15.4f}{max_std:>14.4f}"
f"{parse_failure_rate:>11.0%}{relative_std:>12.2f}x"
)
```
```
condition mean prob std max prob std parse fail x TypeSafe
claude-haiku-4-5 t=0 0.0367 0.2981 0% 3.26x
claude-haiku-4-5 t=default 0.0539 0.3167 2% 4.79x
gpt-5.4-mini t=0 0.0462 0.1929 0% 4.10x
gpt-5.4-mini t=default 0.0626 0.2193 0% 5.56x
gpt-5.5-reasoning 0.0336 0.1337 0% 2.98x
claude-opus-4-8-reasoning 0.0273 0.0733 0% 2.42x
typesafe_choice 0.0113 0.0377 0% 1.00x
```
In this run TypeSafe has a mean probability std dev of `0.0113` and a max single-label std
dev of `0.0377`. The LLM probability rows range from `0.0273` to `0.0626` mean probability
std dev, about `2.4x` to `5.6x` the TypeSafe mean. TypeSafe moves less at the probability
level than every probability-output LLM row here, and still repeats the same picked labels.
## Plot: how often does each condition repeat its own decision?
For each condition and question we find the most common valid label (the plurality pick),
count how many of the 15 draws chose it, and divide by 15. Parse failures count against the
score. A score of `1.0` means the condition landed on its own label every time. The bar
averages that score over all 8 questions.
```python expandable theme={null}
def agreement_rate(samples: list) -> float:
"""Mean over questions of the plurality label's share across all NUM_SAMPLES draws.
Parse failures count against agreement because a failed route is not a repeated decision.
"""
shares = []
for question_key, (_instructions, choices) in QUESTIONS.items():
labels = list(choices)
picks = [
argmax_label(samples[sample][question_key], labels)
for sample in range(NUM_SAMPLES)
]
picks = [pick for pick in picks if pick is not None]
if not picks:
shares.append(0.0)
continue
top = Counter(picks).most_common(1)[0][1]
shares.append(top / NUM_SAMPLES)
return mean(shares) if shares else float("nan")
bar_labels = ALL_LABELS
rates = [agreement_rate(pooled_runs[label]) for label in bar_labels]
def _condition_color(label: str) -> str:
if label == TYPESAFE_LABEL:
return "#2b8cbe" # TypeSafe: blue
return (
"#b30000" if "single-pick" in label else "#fe9929"
) # single-pick red, distribution orange
fig_bar, bar_ax = plt.subplots(figsize=(7, 0.45 * len(bar_labels) + 1))
positions = range(len(bar_labels))
bar_ax.barh(
list(positions),
rates,
color=[_condition_color(label) for label in bar_labels],
alpha=0.85,
)
for position, rate in zip(positions, rates):
bar_ax.text(
rate + 0.01, position, f"{rate:.0%}", va="center", fontsize=8, color="gray"
)
bar_ax.set_yticks(list(positions))
bar_ax.set_yticklabels(bar_labels, fontsize=8)
bar_ax.invert_yaxis() # first condition on top
bar_ax.set_xlim(0, 1.08)
bar_ax.set_xticks(np.linspace(0, 1, 6))
bar_ax.set_xlabel("decision agreement across 15 re-runs (mean over 8 questions)")
for edge in ("top", "right", "left"):
bar_ax.spines[edge].set_visible(False)
bar_ax.tick_params(length=0)
fig_bar.suptitle("How often each condition repeats its own decision", y=1.0)
fig_bar.tight_layout()
display(fig_bar)
```
This chart measures self-agreement, not correctness. Realistic LLM settings score below
`1.0`, which means the same post can route differently on a re-run. Some `temperature=0` or
single-pick rows look perfectly consistent, but that does not make them right: the heatmap
still shows the conditions disagreeing with each other. Extra LLM repeats can paper over
some of the flips, but they do not settle which label is the appropriate one.
## Open it in the TypeSafe playground
The link below opens the same post and rubric in the playground: one post, the same 8
`Choice`s, and TypeSafe `jev-1.12`.
```python theme={null}
playground_link = make_playground_link(
{"post": POST},
{
key: Choice(instructions=instructions, criteria=choices)
for key, (instructions, choices) in QUESTIONS.items()
},
models=[TYPESAFE_MODEL],
)
display(
Markdown(
f"🔗 [Open this post + rubric in the TypeSafe playground]({playground_link})"
)
)
```
Open this post + rubric in the TypeSafe playground →