relay
Wyolet Relay is a self-hostable LLM router that provides a single OpenAI-compatible endpoint in front of multiple providers (OpenAI, Anthropic, Bedrock, Vertex, Azure, Ollama, Groq).
Ficha actualizada 22 sept 2026, 20:54 · upstream 22 sept 2026, 20:28 · release v0.9.19
Qué es relay
Descripción editorial construida a partir del repositorio, su documentación y sus señales de mantenimiento.
Wyolet Relay is a high-throughput, self-hostable LLM router written in Go that sits in front of any provider supporting OpenAI or Anthropic wire formats. It eliminates the need for developers to maintain multiple API keys and endpoints by pooling them behind a single endpoint. The system supports automatic failover between providers, per-key access control through policies, and batch processing where applicable. Each request is metered and logged with full payload capture options available.
The architecture consists of two components: a data plane that accepts inference requests from your application, and a control plane that serves the admin UI and API for configuration. Requests are authenticated using relay keys scoped to specific policies, which define allowed models, providers, and rate limits before routing through an upstream key pool. The catalog of supported models is maintained in versioned repositories, allowing new hosts and models to be added without modifying existing deployments.
Key operational features include disposable, rate-limited API keys that can be minted on-demand; a proxy mode for transparent forwarding of requests from your own infrastructure; and comprehensive observability through Prometheus metrics and structured JSON logging. The system is designed to run at scale with Kubernetes-native deployment patterns, achieving sub-2ms latency per request while supporting thousands of concurrent connections.
- Centralized LLM routing for organizations using multiple AI providers (OpenAI, Anthropic, Azure, etc.) without maintaining separate API keys or endpoints
- Cost optimization through key pooling and automatic failover between providers based on availability and pricing
- Rate limiting enforcement at the application layer to prevent abuse while allowing legitimate burst traffic from different providers
- Fine-grained access control via policies that restrict which models each relay key can reach, enabling multi-tenant isolation
- Batch processing of large inference requests where a provider supports batching (e.g., OpenAI) without requiring code changes
- Transparent proxy mode for organizations with their own infrastructure that need to forward LLM calls through Relay's managed endpoint
- Supports multiple providers including OpenAI, Anthropic, Bedrock, Vertex AI, Azure, Ollama, and Groq under a single unified interface
- Disposable API keys with per-key rate limiting that automatically scale down when usage exceeds configured limits
- Per-policy access control allowing granular restrictions on which models and providers each relay key can reach
- Batch processing capability for providers like OpenAI that natively support batching, reducing network overhead
- Open-source catalog of 400+ pre-seeded models with extensibility to add new hosts or models without deployment changes
- Kubernetes-native design with sub-2ms latency and thousands of requests per second throughput per pod
- Relay does not natively support streaming responses from providers that do not offer them (e.g., some older OpenAI endpoints)
- Batch processing is available only for providers that support it; other providers must be called individually
- The admin UI and API are exposed on port 8081, which may conflict with existing services or require firewall configuration
- Relay does not provide native authentication for external users (e.g., OAuth) — all access is controlled via relay keys
- No built-in retry logic at the application level; failures must be handled by the calling service
Perfil de infraestructura
Estimación inicial derivada del repositorio y de los servicios detectados.
Despliegue detectado
Qué señales reales de instalación hemos encontrado.
Deploy Relay as a Kubernetes pod using `docker run -p 8080:8080 -p 8081:8081 wyolet/relay:standalone`. The admin UI is accessible at http://localhost:8081. To add a provider, create an API key in the Relay dashboard and assign it to one or more policies. For production use, configure TLS certificates on both the relay server and all upstream providers. Enable Prometheus metrics collection for monitoring request rates, costs, and latency. The catalog of models is available at https://github.com/wyolet/relay-catalog; add new hosts there before deploying them into Relay.
Stack y dependencias
Componentes inferidos a partir de manifests, configuración y documentación.
Puede sustituir a
Relaciones detectadas por el análisis del repositorio y su categoría.
Experiencia real de la comunidad
Ayuda a contrastar la estimación automática con instalaciones reales.
Todavía no hay datos comunitarios para este proyecto.