score, which can fall between two levels. The model also returns a probability for every level in probabilities, and a confidence value for the answer.
Example Score questions:
Request structure
The POST request body to the TypeSafe API has the same three top-level fields as any other question type:state, which is the content to evaluate; model; and questions. Each Score question has the following fields:
type: Always"score".instructions: The question the model answers. What it’s rating.criteria: An ordered array of level descriptions, from the low end of the scale to the high end. Needs at least two levels and takes up to 10.
bug_severity in this case. This id is not sent to the model. The answer is returned under the same id.
Levels
Each entry incriteria is a level: one point on the spectrum of possible answers, described in words. A level’s number is its position in the criteria array, starting at 0, so the three entries above are levels 0, 1 and 2. The order of the array is the numbering.
The model gets the descriptions and nothing else, and each level is judged on its own against the state.
The score in the response is a position on the levels spectrum. For a three-level scale it runs from 0 to 2, and it can land between two levels.
Our client SDKs provide typed questions. In Python, the same question is a Score:
system_one method or the https://api.typesafe.ai/v1/systemone endpoint to call a System One model. The model field selects which model handles the request. How to build with TypeSafe covers where in your code to call it.
Use one of our client SDKs or call the TypeSafe API directly. If a coding agent is writing the integration for you, install the TypeSafe agent skill first so it knows the request and response shapes.
instructions and each level in criteria can be a string, an object, or an array. Start with strings. Use an object when a level needs a description plus a few example situations. See Structured level descriptions below and the API reference.Response structure
The response has one entry inanswers per question, under the ids from the request. This is the response to the example request above:
type: The type of TypeSafe question.probabilities: The probability of each level, keyed by level number as a string. The sum of all values is 1.score: The position on the level number line, from 0 to the top level number, which is 2 here. It’s each level number multiplied by its probability, added up: 0 x 0.0 + 1 x 0.70 + 2 x 0.30 = 1.30.legend: Each level number mapped back to its description.confidence: A number from 0 to 1 computed from howprobabilitiesis spread. A single peak on one level means high confidence. Probability spread over several levels means low confidence.
ScoreAnswer has score, confidence, probabilities, and legend as typed fields. The SDK keys probabilities and legend by integer level rather than by string.
Reading a Score
Let’s look at how the score changes with different inputs. For example, using the question and its levels from the request above:probabilities | |||||
|---|---|---|---|---|---|
| State | score | confidence | Level 0 | Level 1 | Level 2 |
| The export button is misaligned by a few pixels on the settings page. | 0.0 | 1.0 | 1.0 | 0.0 | 0.0 |
| The PDF export button does nothing when clicked. I can still export to CSV and convert it myself, but that takes ages. | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 |
| Export to PDF fails with a spinner that never finishes. Some of our team say CSV export still works for them, others say it fails too. | 1.12 | 0.81 | 0.0 | 0.88 | 0.12 |
| The export button crashes the settings page in Safari. It works in Chrome, but a few of our customers only use Safari. | 1.3 | 0.54 | 0.0 | 0.7 | 0.3 |
| Nobody on our team can log in since this morning. We get a 500 error on every attempt. | 2.0 | 1.0 | 0.0 | 0.0 | 1.0 |
probabilities and confidence alongside the score to distinguish these cases.
A fractional score is a position. You can use it to rank reports by severity, or round it to the nearest level when your code needs one outcome. Our entity alignment cookbook shows an example of rounding to the nearest level to make a decision.
Low confidence on a Score usually means one of three things. The levels overlap for this state, the question is measuring more than one thing, or the state doesn’t say enough to place it. Our Confidence docs cover how to use it in your code.
Writing good levels
Describe situations, not degrees. “Broken or degraded feature, but workaround exists” gives the model something to match the state against. “Moderately severe” doesn’t. Concrete descriptions can help the model distinguish levels. Check the answers against known examples; higher confidence alone does not show that a description is better. Every level is evaluated separately. The model doesn’t see a level’s number or its neighbours, so “worse than the previous level” means nothing to it, and numbers in the descriptions or the instructions don’t help. Here is what happens when the levels are only numbers, on the misaligned-button report from the table above:Splitting a complex judgment into several Scores
A complex judgment, one that depends on several things, is best split into one Score per thing. You can then combine the Scores returned from TypeSafe in your code to make the judgment. Some Scores may matter more than others, so give each Score a weight for its relative importance. The weights are yours. When the combined result doesn’t match what your team would decide, change them in code and run again. Send the Scores in one request. They are evaluated in parallel. Adding questions barely changes the response time and costs a few extra question tokens; see Ask multiple questions together. The request below is the spinner ticket from the table above with some more context. It asks three Scores: how severe the bug is, how frustrated the customer is, and how much the report gives an engineer to work with. TypeSafe’s response:severityis 1.24 at confidence 0.63. Same reading as the opening example: the export is broken and some have a workaround.frustrationis 1.45 at confidence 0.33. The wording is civil, but “third time” and “I’m done” shift the score toward the top level, so the model splits 0.55 and 0.45 between “frustrated but civil” and “very angry”. For this ticket the two levels overlap, which explains the low confidence.report_qualityis 3.0 at confidence 1.0. The steps and browser version are both stated.
len(criteria) - 1, to put every score on 0 to 1. Then the weights mean what they say: 0.6 on severity and 0.3 on frustration makes severity count twice as much.
The TypeSafe Python SDK code below asks the three questions, normalizes each score, and combines them using an example priority calculation:
0.6 × 0.62 + 0.3 × 0.725 + 0.1 × 1.0 = 0.6895, which rounds to 0.69.
The weights live in your code, so you can see exactly how the number is made and change it when the ranking doesn’t match what your team would do. If you later need more Scores, add them to TRIAGE_QUESTIONS. The request count stays at one. This technique of breaking a complex judgment into separate Scores and then combining them with weights in your code is called the Composite scoring pattern.
Structured level descriptions
Start with a basic text description for each level. When the model keeps scoring between two neighbouring levels on inputs you think are clear, give each level an object instead of a string, with a field for what the level covers and a field with a few example situations. Use the same field names on every level so the model can compare like with like. The request below is the spinner ticket that we used earlier, but with examples on each level: The response:
In this comparison, the matching example concentrates more probability on one level. The unrelated example changes the result only slightly compared with plain strings. Higher confidence does not establish which answer is correct. Choose examples with known expected levels, then test the revised descriptions on separate inputs before keeping them.