Building a multi-provider AI layer from one codebase

The abstractions and failure handling behind Ollama, Claude, OpenAI, and Grok.

Normalize the common path

Every provider needs a request, a stream, a final response, usage metadata, and an error. A shared interface keeps the assistant and its tests provider-neutral. Adapters translate model-specific messages and tool calls at the boundary instead of spreading conditionals through the product.

Differences still matter

Context windows, streaming events, tool support, rate limits, and safety behavior vary widely. The abstraction must expose capability information and preserve cancellation, while the UI communicates when a selected model cannot perform an operation. If a provider is unavailable, retry only safe transient failures and offer a deliberate fallback rather than silently changing the privacy boundary.

The maintenance lesson is to test the interface with contract fixtures and keep provider-specific behavior small, observable, and replaceable.

Practical follow-up

A provider-neutral test suite should cover streaming completion, tool calls, cancellation, malformed output, and rate-limit responses. Contract fixtures make it possible to add a provider without rewriting the assistant’s core behavior.

Observability belongs at the adapter boundary: record provider, model, latency, token counts, and failure class while excluding prompt content by default. That gives the team enough evidence to improve routing without turning telemetry into another data-exfiltration path.

Conclusion

A provider-neutral layer earns its keep by protecting the application from differences it cannot eliminate. The diagram places one app contract above an adapter layer and several provider implementations. That structure keeps streaming, tools, cancellation, errors, and usage metadata testable at a shared boundary. It does not mean every provider has identical capabilities or failure modes. Capability discovery and deliberate fallback are necessary when a selected model cannot perform an operation. With contract fixtures and adapter-level observability, adding a provider becomes a contained change instead of a rewrite.