How do I get structured JSON out of an LLM API reliably?

To get reliable structured JSON from an LLM API, the key is to not rely on “please output JSON” alone. Use the strongest structured-output features your provider supports, then add validation and retry logic.

Best options, in order

1) Use native structured output / JSON schema support

If your API supports it, this is the most reliable approach.

Common patterns:

• JSON mode: guarantees valid JSON syntax, but not necessarily adherence to your exact schema.
• Structured outputs / schema mode: you provide a schema, and the model is constrained to match it.
• Tool/function calling: model returns arguments for a function, usually well-formed and easier to validate.

If available, prefer:

• Strict schema enforcement
• Enum constraints
• Required fields
• Nullability rules
• No extra properties if possible

2) Define a precise schema

Be explicit about:

• field names
• types
• required vs optional
• allowed values
• nesting
• arrays item types
• whether extra keys are allowed

Example schema idea:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "integer" },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["name", "age"],
  "additionalProperties": false
}

3) Validate every response

Even with structured output features, always validate.

• Parse JSON
• Check schema
• Reject malformed or semantically invalid data
• Retry if needed

Use a validator like:

• JSON Schema validators
• Pydantic / Zod / Marshmallow / AJV

4) Retry with repair prompts

If parsing or validation fails:

• send back the error
• ask the model to fix only the JSON
• keep temperature low

Example repair instruction:

The previous output was invalid because age must be an integer and additionalProperties are not allowed. Return only corrected JSON matching the schema.

5) Keep prompts narrow and deterministic

For structured extraction:

• set temperature = 0 or very low
• don’t ask for prose and JSON in the same response
• say “Return only JSON. No markdown. No commentary.”
• provide examples if helpful

6) Prefer extraction over generation

If you’re extracting from text, make the task simple:

• tell the model exactly what to pull
• map fields explicitly
• avoid ambiguous instructions
• if possible, provide the source text in a clearly delimited block

A robust pattern

Prompt

Extract the following fields from the text and return JSON matching this schema only.
Return no markdown, no explanation, no surrounding text.
If a field is missing, use null.
Schema: …

Runtime flow

1. Call model with schema/JSON mode
2. Parse response
3. Validate against schema
4. If invalid, retry once or twice with correction feedback
5. If still invalid, fail gracefully or fall back to manual handling

Practical tips

• Set temperature: 0
• Set max_tokens high enough for the full JSON
• Use additionalProperties: false if supported
• Avoid deeply nested or overly large schemas if you don’t need them
• Prefer enums over free text when possible
• Use strings for dates only if your schema/validator expects a specific format
• Ask for null rather than omission if your downstream code benefits from fixed keys

If the API does not support schema-constrained output

Then the best fallback is:

• ask for JSON only
• validate
• repair/retry
• optionally use a “JSON parsing” library or constrained decoding layer
• if available, use function calling or structured extraction frameworks

Example workflow in pseudo-code

response = llm(prompt, temperature=0, response_format="json_schema")
data = parse_json(response)
validate(data, schema)

if validation_failed:
    fix_prompt = f"""
    The previous JSON was invalid:
    {validation_error}
    Return only corrected JSON.
    """
    response = llm(fix_prompt, temperature=0, response_format="json_schema")

Bottom line

For reliability:

1. Use schema-constrained output if available
2. Validate the result
3. Retry on failure
4. Keep temperature low
5. Don’t depend on prompt wording alone

If you tell me which API/provider you’re using, I can show the exact way to do it with that SDK.