How can I integrate LLM API providers into a developer tools product with chat and embedding features?

To integrate LLM API providers into a developer tools product that supports both chat and embeddings, design it as a provider-agnostic inference layer with a clean internal API, then map each provider (OpenAI, Anthropic, Google, Azure, Bedrock, etc.) to that interface.

1) Start with a provider abstraction

Create a single internal interface for:

• Chat/completions
• Embeddings
• Model metadata
• Streaming responses
• Tool/function calling
• Rate limits and retries

Example shape:

interface LLMProvider {
  chat(request: ChatRequest): Promise<ChatResponse | AsyncIterable<ChatChunk>>;
  embed(request: EmbedRequest): Promise<EmbedResponse>;
  listModels(): Promise<ModelInfo[]>;
}

This keeps your app logic independent of provider-specific APIs.

---

2) Normalize your data models

Different providers use different request/response formats. Convert everything into your own canonical schema.

Canonical chat request

type ChatRequest = {
  model: string;
  messages: Array<{
    role: 'system' | 'user' | 'assistant' | 'tool';
    content: string | Array<{ type: string; text?: string; imageUrl?: string }>;
  }>;
  temperature?: number;
  maxTokens?: number;
  stream?: boolean;
  tools?: ToolSpec[];
};

Canonical embedding request

type EmbedRequest = {
  model: string;
  input: string | string[];
};

This lets your UI and product features stay consistent across providers.

---

3) Build provider adapters

For each provider, write a thin adapter that translates your canonical schema into the provider’s API.

What each adapter handles

• Auth headers / API keys
• Provider-specific model naming
• Streaming protocol differences
• Tool-calling differences
• Batch embedding formats
• Error normalization

Example responsibilities:

• OpenAI adapter → OpenAI chat/embeddings endpoint
• Anthropic adapter → Claude messages format
• Vertex AI adapter → Google-authenticated endpoints
• Bedrock adapter → AWS auth and model routing

---

4) Separate “model routing” from “provider integration”

Let users choose:

• provider
• model
• region
• fallback chain

Then route requests accordingly.

A simple routing policy:

1. Use the primary provider/model
2. If rate-limited or unavailable, fail over to a backup
3. For embeddings, use a provider with matching vector dimension and cost target

You can support:

• manual provider selection
• cost-based routing
• latency-based routing
• capability-based routing

---

5) Implement chat and streaming

For developer tools, streaming is usually essential.

Recommended approach

• Support server-sent events (SSE) or WebSockets from your backend to the frontend
• Stream tokens/chunks as they arrive from the provider
• Normalize chunks into a single internal event format

Example internal stream events:

type ChatEvent =
  | { type: 'delta'; text: string }
  | { type: 'tool_call'; name: string; arguments: any }
  | { type: 'done' }
  | { type: 'error'; message: string };

This makes the UI easy to build across all providers.

---

6) Build embeddings as a first-class feature

Embeddings are typically used for:

• semantic search
• codebase indexing
• retrieval-augmented generation (RAG)
• similarity matching
• deduplication

Best practices

• Chunk documents before embedding
• Store embeddings in a vector database
• Keep metadata with each vector
• Choose one embedding dimension per index
• Rebuild indexes when model dimensions change

Common vector DBs

• pgvector
• Pinecone
• Weaviate
• Milvus
• Qdrant
• Redis vector search

---

7) Add secrets and tenant isolation

If your product is multi-tenant, handle provider credentials carefully.

Recommended setup

• Store API keys encrypted
• Use per-tenant credentials if customers bring their own keys
• Support workspace-level provider configs
• Never expose raw provider keys to the frontend

Good pattern

• frontend talks to your backend only
• backend holds provider secrets
• backend proxies all LLM requests

---

8) Handle reliability concerns

LLM APIs vary widely in behavior, so your integration layer should include:

• retries with exponential backoff
• timeout handling
• circuit breakers
• provider-specific error normalization
• request id logging
• token/cost tracking
• response caching where appropriate

Normalize errors into a shared format

type LLMError = {
  code: 'rate_limited' | 'auth_failed' | 'invalid_request' | 'provider_down' | 'unknown';
  message: string;
  provider?: string;
  retryable: boolean;
};

---

9) Track usage and billing

For a developer tools product, usage visibility matters.

Track:

• tokens in/out
• embedding calls
• per-provider costs
• latency
• cache hit rate
• per-tenant consumption

Then show:

• usage dashboard
• budgets
• alerts
• quotas
• rate limits per workspace

---

10) Support model capability discovery

Different models support different features:

• chat
• vision
• tool calling
• structured output
• longer context
• embeddings
• JSON mode

Maintain a capabilities registry:

type ModelInfo = {
  provider: string;
  name: string;
  supportsStreaming: boolean;
  supportsTools: boolean;
  supportsEmbeddings: boolean;
  contextWindow: number;
  dimensions?: number;
};

This helps your UI only show valid options.

---

11) Consider a fallback strategy

Because models differ in cost, quality, and uptime, you may want:

• primary model for normal use
• cheaper model for simple tasks
• backup model for outages
• specialized embedding model for search

A common pattern:

• Chat: GPT-4.1 / Claude / Gemini depending on user choice
• Embeddings: text-embedding model from one provider
• Fallbacks: retry provider A, then provider B

---

12) Suggested architecture

A practical architecture looks like this:

• Frontend

  • chat UI
  • prompt builder
  • embedding/search UI
  • usage dashboard

• Backend API

  • auth
  • workspace settings
  • request routing
  • provider adapters
  • logging and analytics

• LLM orchestration layer

  • canonical request schema
  • streaming normalization
  • retries/fallbacks
  • tool execution

• Storage

  • vector DB for embeddings
  • relational DB for metadata, chats, usage
  • secrets manager for API keys

---

13) Recommended implementation steps

1. Define canonical chat and embedding schemas
2. Build one provider adapter first
3. Add streaming support
4. Add embeddings pipeline and vector storage
5. Add usage metering
6. Add provider selection/fallbacks
7. Add tenant-specific keys and admin settings
8. Expand to more providers

---

14) Useful design choice: make providers plugins

If you want easy extensibility, treat providers as plugins:

• core app defines the interface
• each provider is a separate module
• enable/disable providers per deployment
• test each adapter independently

This makes adding new vendors much simpler.

---

If you want, I can also provide:

1. a reference architecture diagram,
2. a TypeScript implementation skeleton, or
3. an example using OpenAI + Anthropic + embeddings + vector search.