Parallel questions

You have one document and N questions about it. You can send one request with all N questions, or N requests with one question each. With TypeSafe the answers come out the same either way: each question is scored on its own against the document, so its answer doesn’t depend on what else is in the request. To check that, this cookbook asks both ways several times and compares the run-to-run std dev - how far each answer moves from one repeat to the next. Whatever noise a question type has, it is the same under both batching strategies (all N in one request, or one question per request); batching adds none. Each noul comes back bit-identical across all 5 repeats either way - the same value on every call, std dev exactly 0.0. The one thing that does change is cost and speed. The document dominates every request, so N single-question calls pay for it N times and make N round trips, while the batched call pays once. The bigger the document, the closer the saving gets to a full Nx. The case here is a regulatory briefing. The document is the Wikipedia article on the GDPR (~54,000 characters, a document-dominated workload where the document is most of every request), and a compliance team wants 13 things checked: 8 Nouls, 2 Choices, and 3 Scores.

Setup

pip install ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/

then set TYPESAFE_API_KEY.

import json
import os
import urllib.request
from pathlib import Path
from statistics import mean, stdev
from time import perf_counter

from cooksafe import JsonCache, make_playground_link
from IPython.display import Markdown, display
from typesafe_sdk import Choice, ChoiceAnswer, Noul, NoulAnswer, Score, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
PRICE = (
    0.10,
    0.30,
)  # $ per 1M tokens (input, output); TypeSafe jev-1.12 as of 2026-07, see README
RUNS = 5  # repeats per batching strategy, to estimate each answer's run-to-run std dev
client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=120.0)
json_cache = JsonCache(Path("json_cache.json"))

Fetched as plain text from a pinned revision of the article and cached in json_cache.json next to the API calls, so the document - and its numbers - stay fixed even as the live article gets edited.

WIKIPEDIA_REVISION = 1363040264  # "General Data Protection Regulation", as of 2026-07


@json_cache
def fetch_article(revision_id: int) -> str:
    url = (
        "https://en.wikipedia.org/w/api.php?action=query&format=json"
        f"&prop=extracts&explaintext=1&revids={revision_id}"
    )
    request = urllib.request.Request(
        url, headers={"User-Agent": "typesafe-cookbook/1.0"}
    )
    with urllib.request.urlopen(request) as response:
        pages = json.loads(response.read())["query"]["pages"]
    return next(iter(pages.values()))["extract"]


DOCUMENT = {
    "source": f"https://en.wikipedia.org/?oldid={WIKIPEDIA_REVISION}",
    "text": fetch_article(WIKIPEDIA_REVISION),
}
print(f"{len(DOCUMENT['text']):,} characters")
display(Markdown(f"📄 [Read the pinned Wikipedia revision]({DOCUMENT['source']})"))

53,777 characters

📄 Read the pinned Wikipedia revision

The questions: 8 nouls + 2 choices + 3 scores

One number tracked per answer, by type:

Noul: the probability of “yes”.
Choice: the max prob, the probability on the picked label. criteria maps each label to its meaning.
Score: the score normalized to 0-1, the score divided by the top level. criteria lists the level descriptions, from level 0 up.

QUESTIONS = {
    "breach_72h": Noul(
        instructions="Must a personal data breach be reported to the supervisory authority within 72 hours?"
    ),
    "applies_non_eu": Noul(
        instructions="Does the regulation apply to organisations established outside the EU that offer goods or services to people in the EU?"
    ),
    "dpo_all_orgs": Noul(
        instructions="Must every organisation appoint a Data Protection Officer, regardless of what data it processes?"
    ),
    "pre_ticked_consent": Noul(
        instructions="Can valid consent be obtained through pre-ticked boxes or inactivity?"
    ),
    "right_erasure": Noul(
        instructions="Does the regulation grant individuals a right to erasure of their personal data?"
    ),
    "data_portability": Noul(
        instructions="Does the regulation include a right to data portability?"
    ),
    "us_federal_law": Noul(instructions="Is the GDPR a United States federal law?"),
    "criminal_penalties": Noul(
        instructions="Does the GDPR itself impose criminal penalties such as imprisonment?"
    ),
    "instrument_type": Choice(
        instructions="What kind of EU legal instrument is the GDPR?",
        criteria={
            "Regulation": "Directly binding law in all member states, no national implementation needed.",
            "Directive": "Sets goals that member states implement through national law.",
            "Treaty": "An international treaty between states.",
            "Recommendation": "Non-binding guidance.",
        },
    ),
    "max_fine": Choice(
        instructions="What is the maximum administrative fine for the most serious infringements?",
        criteria={
            "TwentyM_or_4pct": "Up to EUR 20 million or 4% of annual worldwide turnover, whichever is greater.",
            "TenM_or_2pct": "Up to EUR 10 million or 2% of annual worldwide turnover, whichever is greater.",
            "FixedCap": "A fixed amount not tied to turnover.",
            "NoFines": "The GDPR provides no administrative fines.",
        },
    ),
    "individual_rights": Score(
        instructions="How strong are the rights the GDPR grants to individuals over their data?",
        criteria=[
            "None: individuals get no rights over their data.",
            "Weak: a right to be informed, but little control.",
            "Moderate: access and correction rights, but limited means to act on them.",
            "Strong: access, erasure, portability, and objection rights, with enforcement behind them.",
        ],
    ),
    "penalty_severity": Score(
        instructions="How severe are the penalties the GDPR provides for non-compliance?",
        criteria=[
            "None: no penalties of any kind.",
            "Symbolic: small fixed fines unlikely to change behavior.",
            "Substantial: fines large enough to matter to most companies.",
            "Severe: fines scaled to global revenue, material even to the largest companies.",
        ],
    ),
    "compliance_burden": Score(
        instructions="How heavy is the compliance burden the GDPR places on organisations?",
        criteria=[
            "Negligible: no meaningful obligations.",
            "Light: a few notices and disclosures.",
            "Moderate: documented processes and some dedicated roles for larger processors.",
            "Heavy: records, impact assessments, officers, and breach procedures for many organisations.",
            "Extreme: obligations so demanding that ordinary organisations cannot fully comply.",
        ],
    ),
}
N = len(QUESTIONS)
METRIC = {  # question type -> the one number we track per answer
    Noul: "p(yes)",
    Choice: "max prob",
    Score: "normalized score",
}

Ask two ways, 5 times each

ask() sends any subset of the questions with the document and reduces each answer to its one tracked number. The document is byte-identical in every call. Both batching strategies run RUNS = 5 times, giving each question 5 answers per strategy, enough to compare the mean (do the two agree?) and the std dev (does batching add noise?). Calls are cached to json_cache.json, which ships with the cookbook, so re-rendering is free; delete it to re-run live.

@json_cache
def ask(keys: tuple[str, ...], run: int):
    """One TypeSafe call -> ({key: tracked metric}, cost_usd, latency_s); ``run`` only forces a
    distinct live call per repeat."""
    started = perf_counter()
    response = client.system_one(
        state={"article": DOCUMENT},
        questions={key: QUESTIONS[key] for key in keys},
        model=TYPESAFE_MODEL,
    )
    values = {}
    for key in keys:
        answer = response.answers[key]
        if isinstance(answer, NoulAnswer):
            values[key] = answer.noul
        elif isinstance(answer, ChoiceAnswer):
            values[key] = max(answer.probabilities.values())
        else:
            values[key] = answer.score / (len(QUESTIONS[key].criteria) - 1)
    cost = (
        response.usage.input_tokens / 1e6 * PRICE[0]
        + response.usage.output_tokens / 1e6 * PRICE[1]
    )
    return values, cost, perf_counter() - started


batched = [
    ask(tuple(QUESTIONS), run) for run in range(RUNS)
]  # all N in one call, x RUNS
singles = [
    {key: ask((key,), run) for key in QUESTIONS} for run in range(RUNS)
]  # N x 1, x RUNS

Batching doesn’t change the answers

Per question: the mean and std dev of its tracked number over the 5 runs, under each batching strategy. If batching changed the answers, the batched columns would differ from the single columns - a shifted mean (bias) or a larger std dev (noise).

print(
    f"{'question':<22}{'metric':<18}{'batched mean':>13}{'single mean':>12}"
    f"{'batched std':>13}{'single std':>12}"
)
for key, question in QUESTIONS.items():
    batched_values = [values[key] for values, _cost, _latency in batched]
    single_values = [singles[run][key][0][key] for run in range(RUNS)]
    print(
        f"{key:<22}{METRIC[type(question)]:<18}{mean(batched_values):>13.3f}"
        f"{mean(single_values):>12.3f}{stdev(batched_values):>13.4f}{stdev(single_values):>12.4f}"
    )

question              metric             batched mean single mean  batched std  single std
breach_72h            p(yes)                    0.890       0.890       0.0000      0.0000
applies_non_eu        p(yes)                    0.990       0.990       0.0000      0.0000
dpo_all_orgs          p(yes)                    0.040       0.040       0.0000      0.0000
pre_ticked_consent    p(yes)                    0.030       0.030       0.0000      0.0000
right_erasure         p(yes)                    0.970       0.970       0.0000      0.0000
data_portability      p(yes)                    0.990       0.990       0.0000      0.0000
us_federal_law        p(yes)                    0.010       0.010       0.0000      0.0000
criminal_penalties    p(yes)                    0.110       0.110       0.0000      0.0000
instrument_type       max prob                  1.000       1.000       0.0000      0.0000
max_fine              max prob                  1.000       0.998       0.0000      0.0045
individual_rights     normalized score          0.996       0.998       0.0015      0.0018
penalty_severity      normalized score          0.999       0.999       0.0018      0.0015
compliance_burden     normalized score          0.748       0.749       0.0011      0.0014

Reading the table by question type:

Nouls come back bit-identical across the 5 repeats: std dev exactly 0.0 under both batching strategies, every batched and single call returning the same probability. One call with N nouls gives the same answers as N calls with one noul.
Choices and scores carry a little run-to-run sampling noise, and it’s the same size under both batching strategies, with the means agreeing to within that noise. The noise is a property of the question type, not of how you batch: batching neither shifts the answer nor adds variance.

Either way, there is no batching effect: no question’s answer depends on the 12 other questions sharing its request.

The only difference: cost and speed

Same answers, different bill. The ~54,000-character article dominates every request, so:

Cost (the robust number): the 13 single-question calls re-send the article 13 times; the batched call sends it once. This saving holds however you fire the calls.
Speed: the figure sums the 13 single-call latencies, so it assumes they run one after another. Fire them concurrently and the gap shrinks, but the 13x token cost stays.

Costs and latencies are averaged over the 5 runs and cached alongside the answers.

batched_cost = mean(cost for _values, cost, _latency in batched)
batched_latency = mean(latency for _values, _cost, latency in batched)
singles_cost = mean(
    sum(singles[run][key][1] for key in QUESTIONS) for run in range(RUNS)
)
singles_latency = mean(
    sum(singles[run][key][2] for key in QUESTIONS) for run in range(RUNS)
)
print(f"{'batching':<24}{'calls':>6}{'cost':>12}{'total time':>12}")
print(
    f"{f'one call, all {N}':<24}{1:>6}{'$' + format(batched_cost, '.6f'):>12}{format(batched_latency, '.2f') + 's':>12}"
)
print(
    f"{f'{N} calls, one each':<24}{N:>6}{'$' + format(singles_cost, '.6f'):>12}{format(singles_latency, '.2f') + 's':>12}"
)
print(
    f"\nbatching: {singles_cost / batched_cost:.1f}x cheaper, {singles_latency / batched_latency:.1f}x faster"
)

batching                 calls        cost  total time
one call, all 13             1   $0.001207       0.31s
13 calls, one each          13   $0.013861       2.95s

batching: 11.5x cheaper, 9.6x faster

Open it in the TypeSafe playground

The same article and the same 13 questions, packed into a share link. Open it to re-run the briefing live; the same numbers come back.

playground_link = make_playground_link(
    {"article": DOCUMENT}, QUESTIONS, models=[TYPESAFE_MODEL]
)
display(
    Markdown(
        f"🔗 [Open this article + questions in the TypeSafe playground]({playground_link})"
    )
)

Open this article + questions in the TypeSafe playground →

Introduction

TypeSafe concepts

TypeSafe foundations

Patterns

Demos

Client SDKs

Reference

Cookbooks

Setup

The questions: 8 nouls + 2 choices + 3 scores

Ask two ways, 5 times each

Batching doesn’t change the answers

The only difference: cost and speed

Open it in the TypeSafe playground

​Setup

​The document: the Wikipedia article on the GDPR

​The questions: 8 nouls + 2 choices + 3 scores

​Ask two ways, 5 times each

​Batching doesn’t change the answers

​The only difference: cost and speed

​Open it in the TypeSafe playground

Setup

The document: the Wikipedia article on the GDPR

The questions: 8 nouls + 2 choices + 3 scores

Ask two ways, 5 times each

Batching doesn’t change the answers

The only difference: cost and speed

Open it in the TypeSafe playground