InferHub
Self-hosted Ollama-compatible inference mesh in .NET with coordinator and GPU worker nodes.
Ficha actualizada 23 sept 2026, 0:24 · upstream 23 sept 2026, 0:17 · release v3.48.0
Qué es InferHub
Descripción editorial construida a partir del repositorio, su documentación y sus señales de mantenimiento.
InferHub is a self-hosted, Ollama-compatible inference mesh that runs the gateway where you have no GPU and workers on machines with GPUs. It closes the gap between always-on small VPS servers and powerful desktops or workstations behind home routers. The coordinator exposes an Ollama-style API while nodes handle actual generation via their local inference backends, all orchestrated through a single stable address without port forwarding.
The architecture consists of a lightweight ASP.NET Core coordinator that authenticates remote callers with Bearer tokens, maintains model availability information, and routes requests to capable GPU nodes. Each node runs an LLM backend abstraction (currently Ollama) behind a small interface, opens persistent outbound connections to the coordinator for reliable communication, and streams results back in real time.
Key design decisions include pluggable backends that can be swapped without touching the coordinator, asynchronous job handling with per-step progress tracking, cooperative cancellation semantics, and a clean separation between model serving and tool execution. The system is designed so existing Ollama clients and scripts continue working unchanged while gaining access to a pool of GPU resources.
- Run LLM inference on a small VPS without any GPU hardware
- Offload heavy generation tasks from powerful desktops or workstations to dedicated nodes
- Deploy an Ollama-compatible API for external clients while keeping the actual model serving local
- Create a distributed inference pipeline where multiple GPUs serve different models simultaneously
- Run AI applications behind a home router without exposing GPU ports directly to the internet
- Build a multi-node inference cluster that can scale horizontally based on available GPU capacity
- Supports Ollama-style API with Bearer token authentication for seamless client compatibility
- Pluggable backend architecture allows swapping between different LLM providers without redeploying the coordinator
- Asynchronous job handling with per-step progress tracking and cooperative cancellation semantics
- Zero new dependencies — PyTorch is a child process, not an external package
- Supports multiple modalities including text-to-image (Stable Diffusion), speech (Whisper/Piper), and video generation via OpenAI's API
- Full observability with Prometheus metrics endpoint and comprehensive console dashboard
- Requires a minimum of one GPU node to serve any inference workload
- Each node requires its own Ollama instance or compatible backend for serving models
- The coordinator must remain reachable from all nodes in the mesh
- No built-in model versioning or rollback capability beyond what Ollama provides at the client level
- Worker nodes are not designed to serve requests directly — they route through the coordinator
Perfil de infraestructura
Estimación inicial derivada del repositorio y de los servicios detectados.
Despliegue detectado
Qué señales reales de instalación hemos encontrado.
Deploy InferHub by creating a .NET project targeting .NET 10.0, adding the required NuGet packages (OllamaClient, ASP.NET Core Web API), and configuring environment variables for database connection strings, authentication tokens, and model backends. Deploy the coordinator as an ASP.NET Core web application on any machine without GPU access, then deploy worker nodes that pull the coordinator's configuration and connect via persistent outbound connections to its HTTP endpoints. For production use, consider using Docker containers with a managed service or Kubernetes for orchestration. The coordinator should be placed in a dedicated subnet to avoid routing conflicts, and all node instances should have consistent network configurations.
Stack y dependencias
Componentes inferidos a partir de manifests, configuración y documentación.
Puede sustituir a
Relaciones detectadas por el análisis del repositorio y su categoría.
Experiencia real de la comunidad
Ayuda a contrastar la estimación automática con instalaciones reales.
Todavía no hay datos comunitarios para este proyecto.