Asking for structure: JSON mode, tool schemas, structured outputs

Your database has typed columns. Your queue expects a known message format. The model returns text. Somewhere between the two, text has to become data, and that step is where most LLM features break in production.

ShipFast's first classifier asked, in plain words, "Reply in JSON with the fields intent and reason." On 1,000 real messages, 974 replies parsed. The other 26 failed in creative ways: the JSON was wrapped in a Markdown code fence, or began with "Sure! Here is the JSON:", or had a trailing comma, or used the key Intent instead of intent. A 2.6% failure rate sounds small until you multiply it by 40,000 messages a day: over a thousand messages stuck every day.

There are better ways to ask. This lesson compares them and picks one for ShipFast.

Four ways to ask for JSON

1. Ask in the prompt — "Reply with JSON only." No guarantee at all. Works most of the time, fails in the ways above.
2. JSON mode — a provider setting (for example OpenAI's response_format={"type": "json_object"}) that guarantees syntactically valid JSON. It does not guarantee your keys, types or allowed values.
3. Tool schemas — you describe a "tool" with a JSON schema for its input and ask the model to call it. The arguments follow the schema closely, and with strict mode they follow it exactly. It was the common trick before providers offered a direct option.
4. Structured outputs — you send a JSON schema with the request and the provider constrains generation so the reply always matches it. This is the direct route when all you want is JSON.

The last column never changes. Nothing the provider offers can guarantee that "tomorrow" was turned into the right date, or that the tracking ID the model copied is the one the customer wrote. That is your validation's job, covered in the next lesson.

Structured outputs in ShipFast

ShipFast uses structured outputs. You already describe your data with Pydantic, and Pydantic can produce a JSON schema, so the schema comes for free.

1from shipfast.schemas import Classification23schema = Classification.model_json_schema()4# {'properties': {'reason': {'type': 'string', ...},5#                 'intent': {'$ref': '#/$defs/Intent'}},6#  'required': ['reason', 'intent'], 'additionalProperties': False, ...}

LLMClient.complete() gains one optional argument, schema. This is the one change to the client in this section, and it stays inside the adapter because every provider spells it differently.

1# shipfast/llm.py, inside class LLMClient (replaces the earlier complete)2    async def complete(self, *, feature: str, system: str, messages: list[dict],3                       max_tokens: int = 512, schema: dict | None = None) -> LLMResult:4        output_config: dict[str, Any] = {}5        if self.effort:6            output_config["effort"] = self.effort    # how hard the model thinks first7        if schema is not None:8            output_config["format"] = {9                "type": "json_schema", "schema": anthropic.transform_schema(schema)}10        start = time.perf_counter()11        try:12            resp = await self._sdk.messages.create(13                model=self.model, max_tokens=max_tokens, system=system,14                messages=messages, output_config=output_config or anthropic.omit)15        except anthropic.APIError as err:16            raise _translate(feature, err) from err17        # ... the rest is unchanged: build LLMResult, log, return

anthropic.transform_schema adapts a normal JSON schema to what the API accepts. Providers support a subset of JSON Schema. Constraints such as minimum lengths, numeric ranges and regex patterns are usually not enforced during generation, so the helper moves them into field descriptions. That is fine: the model still reads them as guidance, and Pydantic enforces them afterwards.

For orientation, here is the same request with OpenAI's Responses API. The schema goes into text.format, and strict turns on enforcement.

1from openai import OpenAI23# SYSTEM and text are the classifier's instructions and the customer's message4response = OpenAI().responses.create(5    model="gpt-5-mini",6    instructions=SYSTEM,7    input=f"<message>\n{text}\n</message>",8    text={"format": {"type": "json_schema", "name": "classification",9                     "schema": Classification.model_json_schema(), "strict": True}},10)11label = Classification.model_validate_json(response.output_text)

Both providers need the same things from your schema to enforce it strictly: every property listed in required, and additionalProperties set to false. Pydantic produces both when you declare extra="forbid" and give no field a default. You will see this pattern in the next lesson.

Tool schemas, and when they still make sense

Before structured outputs existed, the standard trick was to define a tool such as record_intent with your schema as its input and force the model to call it. The model's "call" was your JSON. It still works, and with strict: true on the tool definition the arguments match the schema exactly.

Use tool schemas when the model is genuinely choosing between actions, for example "look up the parcel" or "answer directly". When you only want JSON back, structured outputs are simpler: no fake tool, no tool-call parsing, and no dependence on forcing a specific tool, which some of the newest models no longer allow.

What it costs

Structured outputs do not change the price per token. They often reduce output tokens a little, because the model cannot add "Sure, here is..." or explanations. Some providers compile a schema the first time they see it, so the first call with a new schema can be slower; later calls with the same schema are not. Keep schemas stable and small, and do not generate a new schema per request.