README.md
Rama main · réplica de lectura del repositorio original.
# Local AI Server for Linux
<div align="center">
<img src="https://raw.githubusercontent.com/hossbit/mirassets/main/images/localai-hero.webp" alt="LocalAI local LLM server" width="900">
</div>
<div align="center">
</div>
Run GGUF language models locally with
[llama.cpp](https://github.com/ggml-org/llama.cpp),
CPU or GPU acceleration, and
[llama-swap](https://github.com/mostlygeek/llama-swap).
The server exposes an OpenAI-compatible API and discovers models placed in
the configured install directory, which defaults to `~/ai/models`.
<div align="center">
<a href="https://github.com/hossbit/local-ai-server/releases/latest"><img src="https://raw.githubusercontent.com/hossbit/mirassets/main/images/localai-badge-release.svg" alt="Latest Release" height="68"></a>
<a href="https://github.com/hossbit/local-ai-server/pulse"><img src="https://raw.githubusercontent.com/hossbit/mirassets/main/images/localai-badge-download.svg" alt="Downloads (recorded repository clones)" height="68"></a>
<a href="https://github.com/hossbit/local-ai-server/stargazers"><img src="https://raw.githubusercontent.com/hossbit/mirassets/main/images/localai-badge-stars.svg" alt="GitHub Stars" height="68"></a>
</div>
The download badge counts recorded repository clones, not unique users or release
asset downloads. It preserves the historical total and refreshes daily traffic
counts through an hourly scheduled workflow, without counting overlapping days twice.
## ❤️ Support
If you find Local AI Server useful, you can support its continued development:
<div align="center">
<a href="https://donate.stripe.com/bJebJ0fzHeNc5F32fhaMU00">
<img src="https://hossbit.github.io/assets/donation-progress.svg" alt="Donation goal progress for the home lab upgrade" width="520">
</a>
<br>
<strong><a href="https://donate.stripe.com/bJebJ0fzHeNc5F32fhaMU00">Donate via Stripe</a></strong>
<br><br>
<a href="https://buymeacoffee.com/mirhh">
<img src="https://raw.githubusercontent.com/hossbit/mirassets/main/images/bmc-button.webp" alt="Buy me a coffee" width="300">
</a>
</div>
⭐ Starring and sharing the repository also helps a lot.
## Why Local AI Server?
| Feature | **Local AI Server** | Ollama | LM Studio | OpenAI / Gemini |
| -------------------------------------- | :-----------------: | :--------------------: | :-----------------: | :-------------: |
| Runs fully locally and privately | ✅ | ✅ | ✅ | ❌ |
| Designed for Linux servers | ✅ | ✅ | ⚠️ Desktop-focused | ❌ |
| Uses your existing GGUF files directly | ✅ | ⚠️ Import required | ✅ | ❌ |
| Automatic multi-model switching | ✅ | ✅ | ✅ | Cloud-managed |
| OpenAI-compatible API | ✅ | ✅ | ✅ | ✅ |
| User-level systemd service | ✅ | ⚠️ Usually system-wide | ❌ | Not applicable |
| Transparent `llama.cpp` configuration | ✅ | ⚠️ Abstracted | ⚠️ GUI-managed | ❌ |
| No API fees | ✅ | ✅ | ✅ | ❌ |
| Pure Bash, no extra runtime | ✅ | ⚠️ Ships Go binary | ⚠️ Electron app | ❌ |
| Readable scripts, easy to audit | ✅ | ⚠️ Compiled binary | ⚠️ GUI app | ❌ |
| Light footprint, fits minimal VPS | ✅ | ⚠️ Moderate | ⚠️ Heavy desktop | ❌ |
### Main Advantage
> **Local AI Server gives Linux users a private, lightweight and transparent way to run multiple GGUF models through one OpenAI-compatible API, with automatic model switching and systemd service management — all in readable Bash scripts with no runtime to install.**
## What it provides
- OpenAI-compatible chat and completion endpoints
- CPU mode plus optional Vulkan, ROCm, OpenVINO, SYCL, or CUDA llama.cpp backends
- Automatic discovery of `.gguf` model files, with per-model auto-tuning for GPU hardware
- Multimodal, speculative-decoding, and Prometheus metrics support
- On-demand model loading and switching through llama-swap
- A systemd user service
- A `localai` command for service, model, update, and uninstall tasks
## Requirements
- Ubuntu, Debian, Fedora, RHEL, or another compatible x86-64 Linux system
- A working CPU install, or a supported GPU/runtime for your selected backend
- `sudo` access during installation
- Enough RAM and VRAM for the model and quantization you choose
The installer downloads current `llama.cpp` and `llama-swap` releases and can
install required packages with `apt-get`, `dnf`, or `yum`.
## Install
One-line install:
```bash
curl -fsSL https://hossbit.github.io/localai/install.sh | bash
```
CPU-only install:
```bash
curl -fsSL https://hossbit.github.io/localai/install.sh | LLAMA_CPP_BACKEND=cpu bash
```
Already installed and want another backend, including CUDA? No need to
re-run the installer:
```bash
localai backend install vulkan
localai backend install rocm
localai backend install cuda
```
The default install directory is `~/ai`. See the wiki's [Install
LocalAI](https://github.com/hossbit/comai-linux-assistant-wiki/blob/main/Local-AI-Service.md#install-localai)
page for custom directories, manual installs, backend selection, and pinned
component versions.
By default LocalAI tracks upstream llama.cpp's `b[NUM]` bleeding-edge builds
(cut on nearly every commit). To track its slower `vX.Y.Z` stable releases
instead, set `LLAMA_CPP_CHANNEL=stable` (see localai.conf).
## CUDA and Switching Backends
Besides the prebuilt CPU/Vulkan/ROCm/OpenVINO/SYCL backends, LocalAI can also
build `llama.cpp` from source against your own NVIDIA CUDA Toolkit. Once
LocalAI is installed, add or switch backends with the CLI -- no need to
re-run the installer, and no redownload or rebuild once a backend has been
installed before:
```bash
localai backend install cuda # build CUDA without switching to it yet
localai switch cuda # switch to it (installs first if needed)
localai switch vulkan # switch back -- instant
localai backend list # see what's installed and which is active
```
See the wiki's [Backend Selection](https://github.com/hossbit/comai-linux-assistant-wiki/blob/main/Local-AI-Service.md#backend-selection)
page for CUDA prerequisites (`nvcc` vs. `nvidia-smi`), choosing a backend
during the initial install, tuning overrides, and Fedora/RHEL notes.
## Add a model
LocalAI discovers GGUF files from:
```text
~/ai/models
```
Download a `.gguf` model from a source such as Hugging Face, then put it in that
directory. After adding or removing models, reload LocalAI so it regenerates
the config and restarts only if it changed, then list the detected models:
```bash
localai reload
localai models
```
For a single-file model, either place the file directly in `~/ai/models`:
```text
~/ai/models/Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf
```
or keep it in its own folder:
```text
~/ai/models/Qwen2.5-Coder-7B-Instruct-Q4_K_M/
`-- Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf
```
For split GGUF models, keep all shards together in one folder. The first shard
must use canonical llama.cpp split naming, such as `00001-of-00003`:
```text
~/ai/models/DeepSeek-V4-Flash-UD-IQ1_M/
|-- DeepSeek-V4-Flash-UD-IQ1_M-00001-of-00003.gguf
|-- DeepSeek-V4-Flash-UD-IQ1_M-00002-of-00003.gguf
`-- DeepSeek-V4-Flash-UD-IQ1_M-00003-of-00003.gguf
```
LocalAI registers only the first shard. `llama.cpp` loads the remaining shards
automatically.
Recommended layout:
```text
~/ai/models/
|-- Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf
|-- Mistral-7B-Instruct-Q4_K_M/
| `-- Mistral-7B-Instruct-Q4_K_M.gguf
`-- DeepSeek-V4-Flash-UD-IQ1_M/
|-- DeepSeek-V4-Flash-UD-IQ1_M-00001-of-00003.gguf
|-- DeepSeek-V4-Flash-UD-IQ1_M-00002-of-00003.gguf
`-- DeepSeek-V4-Flash-UD-IQ1_M-00003-of-00003.gguf
```
If LocalAI warns that files look like non-canonical split fragments, rename the
files to llama.cpp split format or merge them first:
```bash
llama-gguf-split --merge first-fragment.gguf merged-model.gguf
```
Use `localai suggest` after adding large models to get advisory runtime settings
based on your installed models, RAM, backend, and detected GPU memory. It uses
the actual GGUF file size as the base estimate, not an exact parameter-count
formula. Runtime memory also depends on context length, KV cache type, batch
size, backend buffers, and operating-system headroom.
GPU-backed installs auto-tune per-model GPU layers and flash-attention, use
compatible f16 KV cache defaults, and enable self-speculative decoding by
default. Compatible models can use quantized KV cache through per-model overrides. On multi-GPU systems, `LOCALAI_SPLIT_MODE`, `LOCALAI_TENSOR_SPLIT`,
`LOCALAI_MAIN_GPU`, and `LOCALAI_DEVICE` control how models are placed across
devices. See the wiki for per-model overrides (`models.d`), multi-GPU tuning,
MoE CPU offload, reasoning-model tuning, multimodal `--mmproj` setup,
speculative-decoding tuning, LoRA adapters, metrics, and startup preloading.
## Use the server
Per-model overrides in `conf/models.d/<model-id>.conf` can set `THREADS`,
`PARALLEL`, `BATCH_SIZE`, `UBATCH_SIZE`, `JINJA`, `MLOCK`, and `NO_MMAP`.
Each model starts from the global defaults. For embedding models, size
`UBATCH_SIZE` to fit each input; for example, use `BATCH_SIZE=2048` and
`UBATCH_SIZE=2048` for inputs of up to 2048 tokens. Run `localai reload`
after changing settings.
Use `localai suggest MODEL` for hardware advice, or measure the current
configuration with `localai suggest --benchmark MODEL --runs 3 --tokens 128`.
The benchmark excludes a warm-up and reports median request time and token
speed when provided by the engine. Add `--json` to save results for comparison.
It loads the selected model and can unload another model; it does not change
settings. Use `--type completion` for base models without a chat template,
or `--type embedding` / `--type reranking` for those workloads. Add
`--prompt-file FILE` to benchmark your own text.
LocalAI runs GGUF models supported by your installed llama.cpp version.
For arbitrarily named models, set `MODEL_TYPE=chat`, `completion`, `embedding`,
or `reranking` in their override file. `POOLING` can select an embedding pooling
method when the model requires it. Vision models still need their matching
projector; architecture and modality support comes from llama.cpp.
Start LocalAI:
```bash
localai start
localai check
```
The API is available at `http://127.0.0.1:$(cat ~/ai/conf/port)/v1`.
### Web UIs
Two browser UIs are available with nothing extra to install — llama-swap
and llama.cpp both ship one, and LocalAI just wires them up:
- **llama-swap Web UI** — `http://127.0.0.1:PORT/ui`. Shows model status,
lets you load/unload models, and tails logs.
- **llama.cpp chat UI** — `http://127.0.0.1:PORT/upstream/MODEL_ID/`. The
same chat UI `llama-server` serves standalone, proxied per model through
llama-swap (loading the model on first request, like any other API call).
Run `localai ui` to print both URLs, or `localai ui MODEL_ID` for one
model's chat UI directly:
```bash
$ localai ui
llama-swap Web UI (model status, load/unload, logs):
http://127.0.0.1:11435/ui
llama.cpp chat UI for a specific model:
http://127.0.0.1:11435/upstream/MODEL_ID/
(run 'localai ui MODEL_ID', or 'localai models' for exact IDs)
API-key auth is enabled. When the browser prompts for credentials,
leave the username blank and use your saved API key as the password.
Need a new key? Run 'localai key create browser'; the secret is shown once.
$ localai ui Qwen2.5-Coder-7B-Instruct-Q4_K_M
llama.cpp chat UI for Qwen2.5-Coder-7B-Instruct-Q4_K_M:
http://127.0.0.1:11435/upstream/Qwen2.5-Coder-7B-Instruct-Q4_K_M/
```
LocalAI is a Bash project. These commands open llama-swap’s built-in dashboard
and llama.cpp’s chat interface:
```bash
localai ui --help # all destinations and examples
localai ui --chat --open # chat playground
localai ui --models --open # load/unload models
localai ui --logs --open # troubleshoot errors
localai ui --performance --open # system statistics (when metrics are enabled)
localai ui --hardware --open # hardware overview
localai ui --settings --open # dashboard preferences
```
No separate frontend is installed. On a remote machine, use the printed HTTP
dashboard links in a browser that can reach the server.
Newer llama.cpp builds support `SPEC_DRAFT_SAMPLING=probabilistic` in a model's
`models.d` override for `SPEC_TYPE=draft-simple` or `draft-mtp`. The global
`LOCALAI_SPEC_DRAFT_SAMPLING` is empty by default. A requested setting fails with
a useful error on older engines that lack `--spec-draft-sampling`; other draft
modes ignore it. Existing `EXTRA_ARGS` overrides remain available.
Use `localai ui --open` to launch the dashboard in your desktop browser, or
`localai ui --open MODEL_ID` for a model's chat UI. On SSH/headless systems,
use the printed URL from a browser that can reach the server. The dashboard
in llama-swap v257 includes a searchable model picker, responsive chat, and
live generation statistics.
`localai key list` shows key metadata, not the secret. Use your saved key or
create a new one with `localai key create browser`.
Open the printed URL in a browser. If API-key auth is enabled ([API
keys](#api-keys)), the browser's login prompt wants the username left
blank and an active key as the password.
## Service and helper commands
Most users only need these:
| Command | Purpose |
| --- | --- |
| `localai start` | Start the service. |
| `localai stop` | Unload loaded models, then stop the service. |
| `localai restart` | Restart the service. |
| `localai reload` | Rescan models and restart only if `config.yaml` would change; prints added/removed models. |
| `localai status` | Show service, process, API, and port status. |
| `localai check` | Check the API and model list. |
| `localai models` | List installed `.gguf` models and show loaded state when the API is reachable. |
| `localai ui [--open] [MODEL]` | Print a dashboard or model chat URL; optionally open it in your desktop browser. |
| `localai suggest` | Suggest runtime settings from installed model sizes and detected hardware. |
| `localai load MODEL` | Warm one model. |
| `localai unload MODEL` | Release one loaded model. |
| `localai key ...` | Manage API keys (`create`, `list`, `revoke`, `rotate`) — see [API keys](#api-keys). |
| `localai update` | Update installed components. `--check` only reports what is available. |
| `localai rollback` | Switch back to the llama.cpp/llama-swap versions the last update replaced. |
| `localai version` | Show component versions. |
| `localai uninstall` | Remove helper files; models are kept by default. |
### Safe updates
`localai update` always installs the newest upstream llama.cpp build and
llama-swap release; versions are never pinned by default. To keep a working
setup working:
- If a downloaded build does not run on your system, the previous one is put
back.
- After the update, LocalAI restarts and waits for the API. If it does not
become ready, the previous versions are restored and restarted
automatically.
- The versions an update replaced are kept, so `localai rollback` can switch
back later. Run `localai rollback` again to return to the newer versions.
Only one previous version is kept. To delete it after each successful
update instead, set `LOCALAI_KEEP_PREVIOUS=0` in `~/ai/conf/localai.conf`.
- `localai update --check` shows what would change without changing anything.
- `~/ai/conf/installed.json` records what was actually installed: resolved
release, build, backend, download URL and checksum, llama-swap version, OS,
and GPU. Include it when reporting a problem.
Models, `models.d` overrides, API keys, and your `localai.conf` settings are
never touched by an update or a rollback.
## API keys
By default the API has no authentication, matching llama-swap's own default —
fine as long as `LOCALAI_LISTEN_HOST` stays `127.0.0.1` (loopback only, the
default). If you plan to reach it from another machine on your LAN, create at
least one key first.
```bash
localai key create work-laptop # name is just a label; shown once, then masked
localai key list # id, name, created, status, masked fingerprint
localai key revoke <id> # deactivate a key immediately
localai key rotate <id> # issue a replacement, then revoke the old one
```
`create` and `rotate` print the full secret exactly once, right after it's
active — save it immediately, it cannot be shown again:
```
Key created: work-laptop
id: a1b2c3d4e5f6
created: 2026-07-23T16:32:10Z
Save this key now - it will not be shown again:
sk-localai-9f1a2b3c4d5e6f7089...
Use it as a Bearer token:
curl http://127.0.0.1:11435/v1/models \
-H "Authorization: Bearer sk-localai-9f1a2b3c4d5e6f7089..."
```
Once at least one key is active, every request needs it:
```bash
curl http://127.0.0.1:11435/v1/models \
-H "Authorization: Bearer sk-localai-REPLACE_ME"
```
```python
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:11435/v1",
api_key="sk-localai-REPLACE_ME",
)
```
Behavior notes:
- **No active keys**: the API stays unauthenticated (today's default, and what
every existing install keeps doing after an upgrade — nothing changes until
you run `localai key create`).
- **First active key**: every request now needs a valid `Authorization: Bearer`
header.
- **Revoking the last key**: the API goes back to unauthenticated, unless you
set `LOCALAI_REQUIRE_API_KEY=1` in `localai.conf`, which makes config
generation refuse to produce an unauthenticated config at all.
- **Lost key**: there's no recovery; `rotate` the key (or `revoke` it and
`create` a new one).
- Keys live in `conf/api-keys.tsv` (mode `0600`, outside Git). Active keys
are also rendered into `conf/keys.d/keys.yaml` (mode `0600`), each with a
`# name` comment above it, and merged into the running config via
llama-swap's `--config-dir` — `config.yaml` itself never contains a
plaintext key. Manage keys through `localai key ...`, not by hand-editing
either file; `keys.yaml` is removed entirely once no keys are active.
- Creating, revoking, or rotating a key restarts the running service so the
change takes effect immediately (same "only restart if something actually
changed" rule `localai reload` already uses for models).
API keys authenticate requests; they don't encrypt them. For LAN/WAN access,
put a TLS-terminating reverse proxy in front and restrict it with a firewall
— see [Security](#security).
### Optional reliability and MoE settings
Normal installations need no changes. These settings are off by default and
can be added to `conf/models.d/<model-id>.conf` for a single model.
If streaming requests occasionally fail with `502` or `proxy error: EOF`, use:
```bash
DISABLE_KEEP_ALIVES=1
```
This opens a fresh backend connection for each request and requires llama-swap
v263 or newer. It does not change the public API or OpenCode configuration.
For a compatible MoE model with experts offloaded to CPU, you can try a GPU
expert cache:
```bash
CPU_MOE=1
MOE_CACHE_MIB=256
```
The cache uses additional VRAM and requires llama-server support for
`--moe-cache-mib`. Leave space for the context cache and other GPU allocations.
Compare `localai suggest --benchmark <model-id>` before and after changing it;
more cache is not always faster. Dense models do not benefit from this setting.
Set either option to `0` to disable it. Run `localai reload` after editing
to apply the configuration. If your runtime is too old,
LocalAI explains how to update it and preserves the working configuration.
### Request size limit
Normal installations need no changes. If an image, audio, or other large
request receives `413 Request Entity Too Large`, you can increase the limit
with llama-swap v266 or newer. Edit `~/ai/conf/localai.conf` (or the `conf`
folder in your custom install location):
```bash
LOCALAI_MAX_REQUEST_BODY_MB=500
```
Then run `localai reload`. This limits the complete request body, including
encoded attachments, for all models; it does not enable new model capabilities.
Larger requests can use more system memory. Set the value to an empty string
(`LOCALAI_MAX_REQUEST_BODY_MB=""`) to restore the upstream default, currently
250 MB. Older llama-swap versions work normally while this setting is blank;
LocalAI explains how to update if you request an unsupported setting.
## Documentation
- [ComAI And LocalAI](https://github.com/hossbit/comai-linux-assistant-wiki/blob/main/ComAI-and-LocalAI.md)
- [Local AI Service](https://github.com/hossbit/comai-linux-assistant-wiki/blob/main/Local-AI-Service.md)
- [Troubleshooting](https://github.com/hossbit/comai-linux-assistant-wiki/blob/main/Troubleshooting-LocalAI.md)
## Security
The helper scripts bind llama-swap to `127.0.0.1`, so the API is available only
on the local machine by default. Do not expose it to a network without adding
authentication ([`localai key create`](#api-keys)), TLS, and appropriate
firewall rules.
## Credits
This project is built on top of:
- https://github.com/ggml-org/llama.cpp
- https://github.com/mostlygeek/llama-swap
Special thanks to the maintainers and contributors of these projects.
LocalAI focuses on simplifying installation, configuration, model management,
and service deployment for local LLM environments.