Why LLMs Do Not Return Structured Data by Default
Language models are trained to generate plausible next tokens, not to follow data schemas. Their training objective optimises for human-readable, contextually appropriate text. When you ask an LLM to return JSON without any enforcement, it will usually produce something JSON-like, but it will also add prose explanations, wrap the JSON in markdown code blocks, use inconsistent key naming, omit required fields when they seem unimportant, include additional fields not in your schema, and fail gracefully on nothing. The probability of consistent schema compliance across thousands of production requests is low without enforcement. This is not a model deficiency. The models were not trained to be JSON serialisers. Structured output mechanisms exist to bridge this gap, either by constraining the model's generation process directly or by validating and correcting outputs after generation.
JSON Mode and Response Schemas
The most direct mechanism for structured output is using an API-level constraint. OpenAI's JSON mode instructs the model to produce valid JSON as its output. The model still determines the JSON structure, but the output is guaranteed to be parseable JSON rather than prose with JSON embedded in it. A stronger version is the response schema or structured outputs feature available in newer API versions. You provide a JSON schema definition specifying the exact fields, types, and nesting structure required. The model's generation is constrained at the token level to only produce tokens that are consistent with the schema at each position. This provides near-100% schema compliance and eliminates the need for output parsing fallbacks. Anthropic's Claude API supports structured output through tool use: defining a tool that represents the structured output schema and instructing the model to use that tool to return its response. The result is a guaranteed structured JSON object that matches the tool's parameter schema.
Prompt-Based Structural Guidance
Where API-level schema enforcement is not available or not used, prompt-based guidance is the alternative. This means explicitly defining the expected output format in the prompt, providing examples of correctly formatted outputs, and including instructions for handling cases where required information is absent. The technique works reasonably well for well-specified formats and capable models, but compliance degrades as schemas become more complex or inputs become more ambiguous. Common prompt patterns include: specifying the exact JSON structure in the system prompt with field names and types, instructing the model to use specific values like null or an empty string when a field cannot be determined, and providing a few-shot example showing a correctly formatted response alongside a typical input. Output validation at the application layer is essential when using prompt-based approaches: parse the response, validate against your schema, and handle failures through a retry or fallback path rather than assuming compliance.
Validation and Error Handling
Production structured output pipelines require error handling regardless of the enforcement mechanism used. API-level schema enforcement significantly reduces but does not eliminate edge cases. Prompt-based approaches require more robust validation. A production-grade structured output implementation should include schema validation immediately after parsing, with specific error handling for each failure type. Missing required fields may indicate the model could not find the information in the input, which is a valid signal worth capturing. Type mismatches may indicate a model that misunderstood the field semantics. Extra fields should be stripped rather than causing failures. Retry logic with an escalated prompt that feeds the validation error back to the model often resolves transient failures. For high-volume pipelines, logging schema validation failures by type creates a feedback loop for prompt improvement. The failure rate on structured output tasks is itself a quality metric for your prompt design.
Structured Output for Agentic and Multi-Step Workflows
Structured output becomes especially important in agentic workflows where one LLM call's output is the input to the next step. If the first step returns unstructured prose, the second step either needs to parse it (introducing error propagation) or include it raw (consuming context window and reducing reliability). Designing each step in an agentic pipeline to return a well-defined structured response makes the overall system more modular, testable, and debuggable. It also enables strong typing throughout the pipeline: your orchestration code can work with typed objects rather than strings, making schema violations compile-time or test-time errors rather than runtime surprises. For function calling and tool use, structured output is mandatory: the model must return arguments in the exact format expected by the tool implementation. Schema definition quality is the primary reliability lever for tool-calling accuracy.
Structured Output at SpeedMVPs
At SpeedMVPs, structured output design is a standard part of every AI feature delivered. We use API-level schema enforcement where available, implement validation middleware for every LLM response that feeds application logic, and build retry-with-feedback mechanisms for production pipelines. For clients in regulated sectors, structured outputs provide an auditable record of exactly what the AI system determined for each input, which supports GDPR's requirement for meaningful information about automated decision-making and FCA expectations around model documentation. The schemas, validation logic, and error handling code are all included in the delivery handover. Projects are priced from GBP 8,000 with 2-3 week delivery, and all code is owned by the client.