How do I set up a tool-calling framework for structured outputs and conversation state management in an agent app?

A solid way to set this up is to separate your agent app into three layers:

1. Model I/O layer — handles prompts, tool calls, and structured outputs
2. Conversation state layer — stores messages, tool results, and app state
3. Orchestration layer — decides when to call tools, validate outputs, retry, and continue

Below is a practical framework you can adapt.

---

1) Define your output contract first

If you want structured outputs, make the model return JSON that matches a schema.

Example schema for a task planner:

{
  "type": "object",
  "properties": {
    "response": { "type": "string" },
    "next_action": {
      "type": "string",
      "enum": ["answer", "call_tool", "ask_clarification", "finish"]
    },
    "tool_name": { "type": "string" },
    "tool_args": { "type": "object" }
  },
  "required": ["response", "next_action"]
}

In practice, use:

• JSON Schema
• Pydantic / Zod / TypeScript types
• strict validation after every model response

This gives you predictable outputs and makes tool routing easier.

---

2) Represent tools explicitly

Each tool should have:

• a name
• description
• input schema
• execution function
• optional output schema

Example tool definition:

tools = [
    {
        "name": "search_docs",
        "description": "Search internal documentation",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string"}
            },
            "required": ["query"]
        }
    },
    {
        "name": "create_ticket",
        "description": "Create a support ticket",
        "input_schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "priority": {"type": "string", "enum": ["low", "medium", "high"]}
            },
            "required": ["title", "priority"]
        }
    }
]

Then implement tool handlers in your app:

async def search_docs(query: str):
    ...

async def create_ticket(title: str, priority: str):
    ...

---

3) Store conversation state separately from prompts

Don’t rely only on the prompt for memory. Keep a durable state object.

A good state model:

state = {
    "conversation_id": "abc123",
    "messages": [
        {"role": "user", "content": "Help me with onboarding"},
        {"role": "assistant", "content": "Sure — what do you need?"}
    ],
    "slots": {
        "user_name": None,
        "account_id": None,
        "intent": "onboarding"
    },
    "tool_results": [],
    "workflow փուլ": "collecting_info"
}

Store this in:

• Redis for fast session state
• Postgres for persistence
• vector DB only for retrieval memory, not primary state

Best practice:

• messages = transcript
• slots / metadata = structured state
• tool_results = tool history
• workflow state = finite-state-machine-like phase

---

4) Use a tool-calling loop

A typical agent loop looks like this:

1. Append user message to conversation state
2. Send relevant state + tools to the model
3. Model either:
   • returns a structured final answer
   • requests a tool call
4. Execute the tool
5. Append tool result to state
6. Call the model again with updated context
7. Repeat until done

Pseudo-code:

while True:
    model_output = call_model(messages=state["messages"], tools=tools)

    validated = validate_output(model_output)

    if validated["next_action"] == "call_tool":
        tool_result = await run_tool(
            validated["tool_name"],
            validated["tool_args"]
        )
        state["messages"].append({
            "role": "tool",
            "name": validated["tool_name"],
            "content": tool_result
        })
        continue

    if validated["next_action"] in ["answer", "finish"]:
        state["messages"].append({
            "role": "assistant",
            "content": validated["response"]
        })
        break

    if validated["next_action"] == "ask_clarification":
        state["messages"].append({
            "role": "assistant",
            "content": validated["response"]
        })
        break

---

5) Make the model choose between “structured output” and “tool call”

A clean pattern is to force the model to always return one of these:

• a structured answer
• a tool request
• a clarification request

Example response shape:

{
  "next_action": "call_tool",
  "tool_name": "search_docs",
  "tool_args": { "query": "password reset policy" },
  "response": ""
}

Then your app orchestrates the rest.

This is much more robust than letting the model output arbitrary text and trying to infer intent afterward.

---

6) Validate everything

Never trust the model output directly.

Validate:

• schema conformance
• tool name exists
• tool args match schema
• output length / safety constraints

Example validations:

• unknown tool → reject and reprompt
• malformed JSON → retry with correction prompt
• missing required field → ask model to repair output

This makes your agent stable in production.

---

7) Keep memory layered

A good memory setup:

Short-term memory

Current conversation messages in session state.

Working memory

Current extracted facts/slots:

• name
• account ID
• user goal
• pending action

Long-term memory

Persisted summaries, preferences, or retrieval documents.

For example:

• summarize every N turns
• keep only recent message window in prompt
• retrieve older context via search when needed

---

8) Use a state machine for workflows

For multi-step agents, a simple finite state machine helps a lot.

Example states:

• idle
• gathering_info
• tool_pending
• awaiting_confirmation
• completed
• error

This prevents the agent from drifting.

Example:

if state["workflow"] == "gathering_info" and missing_fields:
    ask_clarifying_question()
elif state["workflow"] == "ready":
    call_tool()

---

9) Recommended architecture

A practical production layout:

• API layer: receives user input
• State store: Redis/Postgres
• Agent orchestrator: manages loop and tool routing
• Tool registry: maps tool names to schemas + functions
• Validator: checks model output
• Logger/tracer: stores prompts, tool calls, failures
• Summarizer: compresses old conversation history

---

10) Minimal implementation blueprint

Components

• AgentState
• ToolRegistry
• ModelClient
• OutputValidator
• AgentRunner

Flow

1. Load state
2. Add user message
3. Call model with tools + schema
4. Parse structured output
5. If tool request:
   • execute tool
   • store result
   • loop
6. Else return response
7. Save state

---

11) Practical tips

• Keep tool descriptions short and precise
• Prefer fewer tools with clear boundaries
• Use deterministic schemas
• Summarize conversation periodically
• Log every model decision and tool invocation
• Set max tool-call depth to avoid infinite loops
• Ask for clarification when required inputs are missing

---

12) If you’re using an LLM API with native tool calling

Many modern APIs support:

• tool definitions
• function calling
• structured JSON output modes

If available, use those directly instead of manual parsing. Your orchestration logic still matters, but the API will handle much of the formatting burden.

---

If you want, I can also give you:

1. a Python reference implementation,
2. a TypeScript/Node version, or
3. a LangGraph-style state machine example.