How do I get structured JSON output from an LLM API reliably?

To get reliably structured JSON from an LLM API, don’t rely on “please output JSON” alone. Use a combination of API-level constraints, schema validation, and retry/repair logic.

Best practices

1) Use a schema, not just a prompt

Define the exact shape of the output with:

• JSON Schema
• Pydantic / Zod / Marshmallow models
• function/tool calling schemas

This gives the model a target and gives your app something to validate against.

2) Prefer APIs with native structured output support

Many LLM APIs now support one or more of:

• JSON mode
• Structured Outputs / schema-constrained decoding
• Function calling / tool calling

If available, use those instead of free-form prompting. They’re much more reliable.

3) Make the model output only the data

Keep instructions explicit:

• “Return only valid JSON”
• “No markdown”
• “No code fences”
• “Match this schema exactly”

Example prompt snippet:

You must return only valid JSON matching this schema:
{
  "name": string,
  "age": integer,
  "tags": array[string]
}
No extra text.

4) Validate every response

Never trust the model output blindly. Parse and validate:

• json.loads(...)
• schema validator
• type checks
• required field checks

If validation fails, treat it as a recoverable error.

5) Add retry with corrective feedback

If the output is invalid, retry with the validation error:

• “Your previous output was invalid because age was a string, not an integer. Re-output only valid JSON.”

This often fixes small formatting mistakes.

6) Use constrained decoding when possible

If your stack allows it, constrain generation at decode time rather than post-processing. This is the most reliable method because invalid tokens are never produced.

7) Keep schemas simple

Reliability drops when schemas get complex. Prefer:

• flat objects
• small enums
• bounded arrays
• explicit optional fields

Avoid ambiguous fields like “content” that can vary widely unless necessary.

8) Lower randomness

For structured output, use:

• low temperature
• low top-p

This reduces format drift.

---

Recommended pattern

1. Define schema
2. Ask model to fill it
3. Validate output
4. Retry on failure
5. Optionally repair malformed JSON
6. Log failures for monitoring

---

Example workflow

Python example

import json
from pydantic import BaseModel, ValidationError

class Person(BaseModel):
    name: str
    age: int
    tags: list[str]

raw = llm_call(prompt)  # your API call

try:
    data = json.loads(raw)
    person = Person.model_validate(data)
except (json.JSONDecodeError, ValidationError) as e:
    # retry with correction prompt
    print("Invalid output:", e)

---

If the API supports tool/function calling

Use it. This is often the cleanest approach:

• you define a function schema
• the model returns arguments matching that schema
• your app parses the arguments directly

This is usually more robust than asking for JSON in plain text.

---

What not to do

• Don’t parse JSON from markdown fences without validation
• Don’t assume “valid-looking” output is correct
• Don’t use regex as your primary parser
• Don’t skip retries for production workflows

---

Practical recommendation

If you want the most reliable setup:

• Use function calling or structured outputs
• Validate with a schema
• Retry with error feedback
• Keep temperature low

If you want, I can give you:

1. a provider-agnostic pattern,
2. an OpenAI-specific example, or
3. a Python/TypeScript implementation.