How do I get structured JSON back from a language model API?

To get structured JSON back from a language model API, the best approach is to use the API’s JSON / schema mode if it supports it. If not, you can still prompt for JSON and validate/repair the output on your side.

Best options

1) Use a structured output / JSON schema feature

Many modern LLM APIs support one of these:

• JSON mode: guarantees valid JSON text
• Structured outputs / schema mode: guarantees JSON that matches a schema

Example idea:

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "number" }
  },
  "required": ["name", "age"],
  "additionalProperties": false
}

This is the most reliable way because the model is constrained to the structure you want.

---

2) Prompt the model to output only JSON

If schema mode isn’t available, instruct it very explicitly:

Return only valid JSON. No markdown, no explanation, no code fences.
Use this schema: ...
If information is missing, use null.

Example prompt:

Extract the following from the text and return only valid JSON:
{
  "title": string,
  "author": string,
  "published_year": number|null
}

Text: "..."

Then parse the response as JSON in your application.

---

3) Validate and retry

Even with good prompting, models can sometimes produce:

• extra commentary
• trailing commas
• code fences
• invalid types

So you should:

1. Parse JSON
2. If parsing fails, retry with a stricter prompt
3. Optionally send the malformed output back and ask the model to fix it

---

Practical tips

• Ask for no markdown and JSON only
• Specify exact keys
• Specify types
• Say what to do if a field is unknown: null, empty string, or omit it
• Set temperature=0 or low for more consistent formatting
• Use a parser/validator in your code
• If your API supports it, use function calling or tool calling for even more reliable structured outputs

---

Example of a strong prompt

You are a data extraction system.

Return only valid JSON matching this schema:
{
  "company": string,
  "employees": number,
  "public": boolean,
  "headquarters": string|null
}

Rules:
- Output JSON only
- No markdown
- No explanation
- Use null if unknown

---

Example parsing approach

• Python: json.loads(response_text)
• JavaScript: JSON.parse(responseText)

If parsing fails, reject and retry.

---

If you want, I can show you:

1. a Python example using an LLM API,
2. a JavaScript example, or
3. the best schema-based approach for a specific provider like OpenAI, Anthropic, or Gemini.