How do I set up a chat completions API for an internal assistant that needs function calling and structured outputs?

To set up a Chat Completions API for an internal assistant with function calling and structured outputs, you typically want this architecture:

1. Client app sends a user message to your backend.
2. Backend calls the model with:
   • conversation history
   • tool/function definitions
   • structured output schema, if needed
3. Model either:
   • responds normally, or
   • requests a tool/function call
4. Your backend executes the function
5. Backend sends the tool result back to the model
6. Model returns the final answer in the format you want

1) Basic API shape

Use the chat/completions endpoint with messages and tools.

Example request

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "You are an internal assistant."},
        {"role": "user", "content": "What is our vacation policy?"}
    ],
)
print(response.choices[0].message.content)

---

2) Add function calling

Define your functions as tools. The model can decide when to call them.

Example with a function

response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "You are an internal assistant."},
        {"role": "user", "content": "Check my PTO balance."}
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_pto_balance",
                "description": "Get PTO balance for an employee",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "employee_id": {"type": "string"}
                    },
                    "required": ["employee_id"],
                    "additionalProperties": False
                }
            }
        }
    ],
    tool_choice="auto"
)

If the model wants to call the function, you’ll get a tool_calls entry in the assistant message.

---

3) Execute the tool call on your backend

When the model returns a tool call, parse its arguments, run your internal function, then send the result back.

Example flow

message = response.choices[0].message

if message.tool_calls:
    tool_call = message.tool_calls[0]
    args = tool_call.function.arguments
    # parse args, e.g. JSON
    result = get_pto_balance(employee_id="12345")

    followup = client.chat.completions.create(
        model="gpt-4.1",
        messages=[
            {"role": "system", "content": "You are an internal assistant."},
            {"role": "user", "content": "Check my PTO balance."},
            message,
            {
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": str(result)
            }
        ]
    )
    print(followup.choices[0].message.content)

---

4) Structured outputs

If you need the assistant to return JSON in a strict schema, use structured outputs. This is useful for:

• ticket creation
• form filling
• CRM records
• policy extraction
• classification

Example schema-based output

response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Extract expense report data."},
        {"role": "user", "content": "Lunch with client on 2026-07-18 for $42.50."}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "expense_record",
            "schema": {
                "type": "object",
                "properties": {
                    "date": {"type": "string"},
                    "category": {"type": "string"},
                    "amount": {"type": "number"},
                    "description": {"type": "string"}
                },
                "required": ["date", "category", "amount", "description"],
                "additionalProperties": False
            }
        }
    }
)
print(response.choices[0].message.content)

The model should return JSON matching the schema.

---

5) Using both function calling and structured outputs

A common pattern is:

• Tools for actions/data retrieval
• Structured outputs for the final response

For example:

• user asks: “Create a ticket for printer issue”
• model calls create_ticket
• backend creates ticket
• model returns a structured JSON summary:

  {
    "ticket_id": "INC-1042",
    "status": "created",
    "next_steps": "IT will respond within 4 hours"
  }
  

You can also structure the tool output itself in your own backend, then feed it to the model.

---

6) Recommended architecture for an internal assistant

Backend responsibilities

• authentication / SSO
• permission checks
• audit logging
• tool execution
• redaction of sensitive data
• retry logic
• message/history storage

Typical tool set

• HR lookups
• calendar availability
• document search
• CRM queries
• ticket creation
• database reads

Security tips

• Never let the model directly access databases or internal APIs
• Validate all tool arguments
• Enforce role-based access control before tool execution
• Sanitize tool results before returning them to the model
• Log tool calls for auditability
• Limit the scope of tools per user/session

---

7) Good system prompt example

You are an internal assistant for employees.
Use tools when you need up-to-date or private internal information.
Never invent policy details, balances, or records.
If a required field is missing, ask a clarifying question.
When returning final answers, follow the requested structured format exactly.

---

8) Practical implementation pattern

A robust loop looks like this:

1. Send user message + tool definitions
2. If model asks for a tool:
   • validate arguments
   • execute tool
   • append tool result
   • call model again
3. Continue until the assistant produces final content
4. If needed, request structured JSON output for the final response

---

9) Minimal end-to-end pseudo-code

messages = [
    {"role": "system", "content": "You are an internal assistant."},
    {"role": "user", "content": "What is my PTO balance?"}
]

response = client.chat.completions.create(
    model="gpt-4.1",
    messages=messages,
    tools=[...],
    tool_choice="auto"
)

msg = response.choices[0].message

if msg.tool_calls:
    tool_call = msg.tool_calls[0]
    result = get_pto_balance(employee_id="12345")

    messages.append(msg)
    messages.append({
        "role": "tool",
        "tool_call_id": tool_call.id,
        "content": str(result)
    })

    final = client.chat.completions.create(
        model="gpt-4.1",
        messages=messages,
        response_format={...}  # optional structured output
    )

    print(final.choices[0].message.content)
else:
    print(msg.content)

---

If you want, I can provide:

1. a working Python example,
2. a Node.js example, or
3. a FastAPI/Express backend skeleton for internal assistant function calling + structured outputs.