> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/ggml-org/llama.cpp/llms.txt
> Use this file to discover all available pages before exploring further.

# llama-server

> HTTP server for serving LLM models with OpenAI-compatible API

## Overview

`llama-server` is a fast, lightweight HTTP server for serving LLM models with an OpenAI-compatible API. Built on pure C/C++ with minimal dependencies, it provides enterprise-grade features like parallel decoding, continuous batching, and multi-user support.

## Quick Start

<CodeGroup>
  ```bash Basic server theme={null}
  llama-server -m model.gguf --port 8080
  ```

  ```bash From Hugging Face theme={null}
  llama-server -hf ggml-org/gemma-3-1b-it-GGUF
  ```

  ```bash With GPU offloading theme={null}
  llama-server -m model.gguf -ngl 35 --port 8080
  ```
</CodeGroup>

The server will start on `http://localhost:8080` with a web UI accessible via browser.

## Key Features

* **OpenAI API Compatible**: Drop-in replacement for OpenAI chat completions and embeddings
* **Anthropic Messages API**: Compatible with Claude API format
* **Parallel Decoding**: Multi-user support with continuous batching
* **Multimodal**: Process images and audio through API endpoints
* **Reranking**: Built-in reranking endpoint for search applications
* **Function Calling**: Tool use support for compatible models
* **Speculative Decoding**: Accelerated generation with draft models
* **Web UI**: Built-in interface for testing and debugging

## Server Configuration

### Basic Server Options

<ParamField path="--host" type="string" default="127.0.0.1">
  IP address to bind to. Use `0.0.0.0` to allow external connections.

  Can also bind to a UNIX socket by ending the address with `.sock`.

  Environment: `LLAMA_ARG_HOST`
</ParamField>

<ParamField path="--port" type="integer" default="8080">
  Port to listen on.

  Environment: `LLAMA_ARG_PORT`
</ParamField>

<ParamField path="--path" type="string">
  Path to serve static files from.

  Environment: `LLAMA_ARG_STATIC_PATH`
</ParamField>

<ParamField path="--api-prefix" type="string">
  Prefix path the server serves from (without trailing slash).

  Environment: `LLAMA_ARG_API_PREFIX`
</ParamField>

### Model Loading

<ParamField path="-m, --model" type="string">
  Path to the GGUF model file.

  Environment: `LLAMA_ARG_MODEL`
</ParamField>

<ParamField path="-hf, --hf-repo" type="string">
  Hugging Face repository in format `<user>/<model>[:quant]`.

  Automatically downloads mmproj for multimodal models unless disabled with `--no-mmproj`.

  Example: `unsloth/phi-4-GGUF:q4_k_m`

  Environment: `LLAMA_ARG_HF_REPO`
</ParamField>

<ParamField path="-a, --alias" type="string">
  Model name aliases (comma-separated) to be used by API.

  Environment: `LLAMA_ARG_ALIAS`
</ParamField>

### Parallel Processing

<ParamField path="-np, --parallel" type="integer" default="-1">
  Number of parallel slots (concurrent requests). `-1` means auto.

  Environment: `LLAMA_ARG_N_PARALLEL`
</ParamField>

<ParamField path="-c, --ctx-size" type="integer" default="0">
  Size of the prompt context. `0` loads from model.

  For parallel requests, multiply by number of slots. Example: `-c 16384 -np 4` supports 4 concurrent requests with 4096 context each.

  Environment: `LLAMA_ARG_CTX_SIZE`
</ParamField>

<ParamField path="-cb, --cont-batching" type="boolean" default="true">
  Enable continuous batching (dynamic batching) for efficient parallel processing.

  Environment: `LLAMA_ARG_CONT_BATCHING`
</ParamField>

### Authentication & Security

<ParamField path="--api-key" type="string">
  API key for authentication. Multiple keys can be comma-separated.

  Environment: `LLAMA_API_KEY`
</ParamField>

<ParamField path="--api-key-file" type="string">
  Path to file containing API keys (one per line).
</ParamField>

<ParamField path="--ssl-key-file" type="string">
  Path to PEM-encoded SSL private key for HTTPS.

  Environment: `LLAMA_ARG_SSL_KEY_FILE`
</ParamField>

<ParamField path="--ssl-cert-file" type="string">
  Path to PEM-encoded SSL certificate for HTTPS.

  Environment: `LLAMA_ARG_SSL_CERT_FILE`
</ParamField>

## Usage Examples

### Starting the Server

<Steps>
  <Step title="Basic startup">
    Start with default configuration:

    ```bash theme={null}
    llama-server -m model.gguf --port 8080
    ```

    Access the web UI at `http://localhost:8080`
  </Step>

  <Step title="Multiple concurrent users">
    Support up to 4 concurrent requests:

    ```bash theme={null}
    # 4 slots × 4096 tokens = 16384 total context
    llama-server -m model.gguf -c 16384 -np 4
    ```
  </Step>

  <Step title="Enable speculative decoding">
    Use a draft model for faster generation:

    ```bash theme={null}
    llama-server -m model.gguf -md draft.gguf
    ```
  </Step>
</Steps>

### Docker Deployment

<CodeGroup>
  ```bash CPU-only theme={null}
  docker run -p 8080:8080 -v /path/to/models:/models \
    ghcr.io/ggml-org/llama.cpp:server \
    -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080
  ```

  ```bash With CUDA theme={null}
  docker run -p 8080:8080 -v /path/to/models:/models --gpus all \
    ghcr.io/ggml-org/llama.cpp:server-cuda \
    -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080 --n-gpu-layers 99
  ```
</CodeGroup>

### Docker Compose

```yaml theme={null}
services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_ENDPOINT_METRICS: 1
      LLAMA_ARG_PORT: 8080
```

## API Endpoints

### Health Check

**GET** `/health` or `/v1/health`

Public endpoint (no API key required).

<CodeGroup>
  ```bash cURL theme={null}
  curl http://localhost:8080/health
  ```

  ```json Response (Ready) theme={null}
  {
    "status": "ok"
  }
  ```

  ```json Response (Loading) theme={null}
  {
    "error": {
      "code": 503,
      "message": "Loading model",
      "type": "unavailable_error"
    }
  }
  ```
</CodeGroup>

### Chat Completions (OpenAI Compatible)

**POST** `/v1/chat/completions`

<CodeGroup>
  ```bash cURL theme={null}
  curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -d '{
      "model": "gpt-3.5-turbo",
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"}
      ],
      "temperature": 0.7
    }'
  ```

  ```python Python theme={null}
  import openai

  client = openai.OpenAI(
      base_url="http://localhost:8080/v1",
      api_key="YOUR_API_KEY"
  )

  response = client.chat.completions.create(
      model="gpt-3.5-turbo",
      messages=[
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Hello!"}
      ]
  )

  print(response.choices[0].message.content)
  ```
</CodeGroup>

### Completions (Non-OAI Format)

**POST** `/completion`

Llama.cpp native completion endpoint with extended features.

```bash theme={null}
curl --request POST \
  --url http://localhost:8080/completion \
  --header "Content-Type: application/json" \
  --data '{
    "prompt": "Building a website can be done in 10 simple steps:",
    "n_predict": 128,
    "temperature": 0.8,
    "top_k": 40,
    "top_p": 0.95
  }'
```

### Embeddings

**POST** `/v1/embeddings`

Generate embeddings with embedding models:

```bash theme={null}
llama-server -m embedding-model.gguf --embedding --pooling cls -ub 8192
```

```bash cURL theme={null}
curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input": "The quick brown fox jumps over the lazy dog.",
    "model": "text-embedding-ada-002"
  }'
```

### Reranking

**POST** `/reranking`

Rerank documents for search applications:

```bash theme={null}
llama-server -m reranking-model.gguf --reranking
```

## Advanced Configuration

### Multimodal Support

Serve vision or audio models:

```bash theme={null}
llama-server -m vision-model.gguf \
  -mm vision-projector.gguf \
  --image-max-tokens 1024
```

The `/v1/chat/completions` endpoint accepts images in base64 format:

```json theme={null}
{
  "model": "gpt-4-vision",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
      ]
    }
  ]
}
```

### Monitoring Endpoints

<ParamField path="--metrics" type="boolean" default="false">
  Enable Prometheus-compatible metrics endpoint at `/metrics`.

  Environment: `LLAMA_ARG_ENDPOINT_METRICS`
</ParamField>

<ParamField path="--slots" type="boolean" default="true">
  Expose slot monitoring endpoint for viewing active requests.

  Environment: `LLAMA_ARG_ENDPOINT_SLOTS`
</ParamField>

<ParamField path="--props" type="boolean" default="false">
  Enable POST /props endpoint for changing global properties.

  Environment: `LLAMA_ARG_ENDPOINT_PROPS`
</ParamField>

### Grammar & JSON Schemas

Constrain all outputs with a grammar:

```bash theme={null}
# Custom grammar
llama-server -m model.gguf --grammar-file grammar.gbnf

# JSON output
llama-server -m model.gguf --grammar-file grammars/json.gbnf
```

Clients can also specify grammars per-request in the API.

### Caching & Performance

<ParamField path="--cache-prompt" type="boolean" default="true">
  Enable prompt caching to reuse KV cache from previous requests.

  Environment: `LLAMA_ARG_CACHE_PROMPT`
</ParamField>

<ParamField path="--cache-reuse" type="integer" default="0">
  Minimum chunk size to attempt reusing from cache via KV shifting.

  Requires prompt caching to be enabled.

  Environment: `LLAMA_ARG_CACHE_REUSE`
</ParamField>

<ParamField path="-sps, --slot-prompt-similarity" type="float" default="0.1">
  How much a request prompt must match a slot's prompt to reuse that slot.

  `0.0` disables this feature.
</ParamField>

### Router Mode

Serve multiple models simultaneously:

<ParamField path="--models-dir" type="string">
  Directory containing models for router server.

  Environment: `LLAMA_ARG_MODELS_DIR`
</ParamField>

<ParamField path="--models-max" type="integer" default="4">
  Maximum number of models to load simultaneously. `0` = unlimited.

  Environment: `LLAMA_ARG_MODELS_MAX`
</ParamField>

<ParamField path="--models-autoload" type="boolean" default="true">
  Automatically load models on demand.

  Environment: `LLAMA_ARG_MODELS_AUTOLOAD`
</ParamField>

### Timeout & Throttling

<ParamField path="-to, --timeout" type="integer" default="600">
  Server read/write timeout in seconds.

  Environment: `LLAMA_ARG_TIMEOUT`
</ParamField>

<ParamField path="--threads-http" type="integer" default="-1">
  Number of threads to process HTTP requests.

  Environment: `LLAMA_ARG_THREADS_HTTP`
</ParamField>

<ParamField path="--sleep-idle-seconds" type="integer" default="-1">
  Seconds of idleness before server sleeps to save resources. `-1` disables.
</ParamField>

## Web UI Configuration

<ParamField path="--webui" type="boolean" default="true">
  Enable the built-in web interface.

  Environment: `LLAMA_ARG_WEBUI`
</ParamField>

<ParamField path="--webui-config" type="json">
  JSON configuration for WebUI defaults.

  Environment: `LLAMA_ARG_WEBUI_CONFIG`
</ParamField>

<ParamField path="--webui-config-file" type="string">
  Path to JSON file with WebUI configuration.

  Environment: `LLAMA_ARG_WEBUI_CONFIG_FILE`
</ParamField>

## Environment Variables

Boolean options use these values:

* **Enabled**: `true`, `1`, `on`, `enabled`
* **Disabled**: `false`, `0`, `off`, `disabled`
* **Negation**: `LLAMA_ARG_NO_MMAP` disables mmap regardless of value

Example:

```bash theme={null}
export LLAMA_ARG_MODEL=/models/my_model.gguf
export LLAMA_ARG_CTX_SIZE=4096
export LLAMA_ARG_N_PARALLEL=2
export LLAMA_ARG_ENDPOINT_METRICS=1
export LLAMA_ARG_MMAP=true

llama-server
```

## Performance Optimization

<Note>
  **Best Practices**

  * Use `--cont-batching` for multiple concurrent users
  * Enable `--cache-prompt` to reuse computation across similar requests
  * Set `--cache-reuse` for improved performance with shared prefixes
  * Use `--flash-attn on` on supported hardware
  * Adjust `-np` (parallel slots) based on your concurrency needs
  * Monitor with `--metrics` endpoint for production deployments
</Note>

## Building with SSL

To enable HTTPS support:

```bash theme={null}
cmake -B build -DLLAMA_OPENSSL=ON
cmake --build build --config Release -t llama-server
```

Then use with SSL certificates:

```bash theme={null}
llama-server -m model.gguf \
  --ssl-key-file server-key.pem \
  --ssl-cert-file server-cert.pem \
  --port 8443
```

## See Also

* [llama-cli](/api/tools/llama-cli) - Interactive command-line interface
* [llama-bench](/api/tools/llama-bench) - Performance benchmarking
* [Server Development Guide](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README-dev.md)