> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/ggml-org/llama.cpp/llms.txt
> Use this file to discover all available pages before exploring further.

# Server (llama-server)

> OpenAI-compatible HTTP API server for llama.cpp

The `llama-server` provides a production-ready HTTP API server with OpenAI-compatible endpoints for chat completions, embeddings, and more.

## Features

* OpenAI-compatible `/v1/chat/completions` and `/v1/embeddings` endpoints
* Anthropic Messages API compatibility
* Parallel decoding with multi-user support
* Continuous batching for optimal throughput
* Multimodal support (vision and audio)
* Web UI for interactive testing
* Reranking endpoint
* Function calling / tool use
* Speculative decoding

## Quick Start

<Steps>
  <Step title="Start the server">
    Launch llama-server with your model:

    ```bash theme={null}
    ./llama-server -m models/model.gguf -c 2048
    ```

    The server will listen on `http://127.0.0.1:8080` by default.
  </Step>

  <Step title="Access the Web UI">
    Open your browser and navigate to:

    ```
    http://127.0.0.1:8080
    ```
  </Step>

  <Step title="Test the API">
    Make a request to the chat completions endpoint:

    ```bash theme={null}
    curl http://localhost:8080/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
        "model": "gpt-4",
        "messages": [{"role": "user", "content": "Hello!"}]
      }'
    ```
  </Step>
</Steps>

## Starting the Server

### Basic Configuration

```bash theme={null}
# Start with default settings
./llama-server -m models/model.gguf

# Custom host and port
./llama-server -m models/model.gguf --host 0.0.0.0 --port 8080

# With GPU acceleration
./llama-server -m models/model.gguf -ngl 99
```

### Docker

```bash theme={null}
# Basic Docker run
docker run -p 8080:8080 -v /path/to/models:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  -m models/model.gguf -c 512 --host 0.0.0.0 --port 8080

# With CUDA GPU support
docker run -p 8080:8080 -v /path/to/models:/models --gpus all \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m models/model.gguf -c 512 --host 0.0.0.0 --port 8080 --n-gpu-layers 99
```

### Docker Compose

```yaml theme={null}
services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_ENDPOINT_METRICS: 1
      LLAMA_ARG_PORT: 8080
```

<Note>
  For boolean environment variables like `LLAMA_ARG_MMAP`, use values: `true`/`1`/`on`/`enabled` or `false`/`0`/`off`/`disabled`
</Note>

## Chat Completions API

OpenAI-compatible endpoint at `/v1/chat/completions`.

### Basic Request

```bash theme={null}
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'
```

### Streaming

```bash theme={null}
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Count to 5"}],
    "stream": true
  }'
```

### With Temperature and Top-P

```bash theme={null}
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Write a poem"}],
    "temperature": 0.9,
    "top_p": 0.95,
    "max_tokens": 200
  }'
```

### JSON Mode

```bash theme={null}
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Generate a user profile"}],
    "response_format": {"type": "json_object"}
  }'
```

<CodeGroup>
  ```python Python theme={null}
  import openai

  client = openai.OpenAI(
      base_url="http://localhost:8080/v1",
      api_key="not-needed"
  )

  response = client.chat.completions.create(
      model="gpt-4",
      messages=[
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Hello!"}
      ]
  )

  print(response.choices[0].message.content)
  ```

  ```javascript Node.js theme={null}
  import OpenAI from 'openai';

  const openai = new OpenAI({
    baseURL: 'http://localhost:8080/v1',
    apiKey: 'not-needed'
  });

  const response = await openai.chat.completions.create({
    model: 'gpt-4',
    messages: [
      { role: 'system', content: 'You are a helpful assistant.' },
      { role: 'user', content: 'Hello!' }
    ]
  });

  console.log(response.choices[0].message.content);
  ```

  ```bash cURL theme={null}
  curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "gpt-4",
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"}
      ]
    }'
  ```
</CodeGroup>

## Completion API

Non-OpenAI-compatible endpoint at `/completion` for raw text completion.

```bash theme={null}
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Building a website can be done in 10 simple steps:",
    "n_predict": 128,
    "temperature": 0.7,
    "top_p": 0.9
  }'
```

### With Streaming

```bash theme={null}
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Once upon a time",
    "stream": true,
    "n_predict": 100
  }'
```

### Multiple Prompts

```bash theme={null}
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": ["First prompt", "Second prompt", "Third prompt"],
    "n_predict": 50
  }'
```

## Server Configuration

### Parallel Processing

```bash theme={null}
# Support multiple simultaneous users (default: auto)
./llama-server -m model.gguf -np 4

# Enable continuous batching (default: enabled)
./llama-server -m model.gguf -cb
```

### Context and Caching

```bash theme={null}
# Set context size
./llama-server -m model.gguf -c 4096

# Enable prompt caching (default: enabled)
./llama-server -m model.gguf --cache-prompt

# Enable KV cache reuse via shifting
./llama-server -m model.gguf --cache-reuse 256
```

### Batch Processing

```bash theme={null}
# Configure batch sizes for throughput
./llama-server -m model.gguf -b 2048 -ub 512
```

## Authentication

### API Key

```bash theme={null}
# Single API key
./llama-server -m model.gguf --api-key "your-secret-key"

# Multiple API keys (comma-separated)
./llama-server -m model.gguf --api-key "key1,key2,key3"

# Load keys from file
./llama-server -m model.gguf --api-key-file keys.txt
```

### Making Authenticated Requests

```bash theme={null}
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}'
```

## SSL/TLS Configuration

```bash theme={null}
# Build with SSL support
cmake -B build -DLLAMA_OPENSSL=ON
cmake --build build --config Release -t llama-server

# Run with SSL certificates
./llama-server -m model.gguf \
  --ssl-key-file server-key.pem \
  --ssl-cert-file server-cert.pem
```

## Monitoring Endpoints

### Health Check

```bash theme={null}
# Check if server is ready
curl http://localhost:8080/health

# Response when ready:
# {"status": "ok"}

# Response when loading:
# {"error": {"code": 503, "message": "Loading model", "type": "unavailable_error"}}
```

### Slots Monitoring

```bash theme={null}
# View slot status (enabled by default)
curl http://localhost:8080/slots
```

### Prometheus Metrics

```bash theme={null}
# Enable metrics endpoint
./llama-server -m model.gguf --metrics

# Access metrics
curl http://localhost:8080/metrics
```

### Properties

```bash theme={null}
# Get server properties
curl http://localhost:8080/props

# Enable POST to modify properties
./llama-server -m model.gguf --props
```

## Model Management

### Model Aliases

```bash theme={null}
# Set model aliases for API routing
./llama-server -m model.gguf -a "gpt-4,gpt-4-turbo"
```

### Model Tags

```bash theme={null}
# Add informational tags
./llama-server -m model.gguf --tags "instruct,chat,7b"
```

### Router Server Mode

```bash theme={null}
# Serve multiple models from a directory
./llama-server --models-dir ./models --models-max 4

# With model presets
./llama-server --models-dir ./models --models-preset presets.ini

# Disable autoload
./llama-server --models-dir ./models --no-models-autoload
```

## Advanced Features

### Sleeping on Idle

```bash theme={null}
# Put server to sleep after idle period (saves resources)
./llama-server -m model.gguf --sleep-idle-seconds 300
```

### Slot Prompt Similarity

```bash theme={null}
# Reuse slots with similar prompts (0.0-1.0, 0.0 = disabled)
./llama-server -m model.gguf -sps 0.5
```

### Context Checkpoints

```bash theme={null}
# Set max checkpoints per slot for state saving
./llama-server -m model.gguf --ctx-checkpoints 8
```

### Slot Persistence

```bash theme={null}
# Save slot KV cache to disk
./llama-server -m model.gguf --slot-save-path ./cache
```

### Static File Serving

```bash theme={null}
# Serve static files from directory
./llama-server -m model.gguf --path ./public

# Disable Web UI
./llama-server -m model.gguf --no-webui

# Custom Web UI config
./llama-server -m model.gguf --webui-config-file config.json
```

## Utility Endpoints

### Tokenization

```bash theme={null}
# Tokenize text
curl http://localhost:8080/tokenize \
  -H "Content-Type: application/json" \
  -d '{"content": "Hello world!"}'

# With token pieces
curl http://localhost:8080/tokenize \
  -H "Content-Type: application/json" \
  -d '{"content": "Hello world!", "with_pieces": true}'
```

### Detokenization

```bash theme={null}
# Convert tokens back to text
curl http://localhost:8080/detokenize \
  -H "Content-Type: application/json" \
  -d '{"tokens": [123, 456, 789]}'
```

### Apply Chat Template

```bash theme={null}
# Format messages using model's chat template
curl http://localhost:8080/apply-template \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are helpful."},
      {"role": "user", "content": "Hello!"}
    ]
  }'
```

## Code Infill

For code completion models with fill-in-the-middle (FIM) support:

```bash theme={null}
curl http://localhost:8080/infill \
  -H "Content-Type: application/json" \
  -d '{
    "input_prefix": "def fibonacci(n):\n    ",
    "input_suffix": "\n    return result",
    "prompt": "# Calculate fibonacci"
  }'
```

### With Repository Context

```bash theme={null}
curl http://localhost:8080/infill \
  -H "Content-Type: application/json" \
  -d '{
    "input_prefix": "def process_data():\n    ",
    "input_suffix": "\n    return result",
    "input_extra": [
      {"filename": "utils.py", "text": "def helper(): ..."},
      {"filename": "config.py", "text": "API_KEY = ..."}
    ]
  }'
```

## LoRA Adapters

### Loading LoRA

```bash theme={null}
# Load LoRA adapters at startup
./llama-server -m model.gguf --lora adapter1.gguf,adapter2.gguf

# With scaling
./llama-server -m model.gguf --lora-scaled "adapter1.gguf:0.5,adapter2.gguf:1.0"

# Load without applying (apply via API)
./llama-server -m model.gguf --lora adapter.gguf --lora-init-without-apply
```

### Managing LoRA via API

```bash theme={null}
# Apply LoRA to specific request
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Specialized task",
    "lora": [{"id": 0, "scale": 0.8}, {"id": 1, "scale": 1.2}]
  }'
```

<Note>
  Requests with different LoRA configurations won't be batched together, which may affect throughput.
</Note>

## Performance Tuning

### Threading

```bash theme={null}
# Set threads for generation and batch processing
./llama-server -m model.gguf -t 8 -tb 16

# HTTP request threads
./llama-server -m model.gguf --threads-http 4
```

### GPU Configuration

```bash theme={null}
# Offload layers to GPU
./llama-server -m model.gguf -ngl 99

# Split across multiple GPUs
./llama-server -m model.gguf -ngl 99 -sm layer -ts 3,1

# Specify devices
./llama-server -m model.gguf -dev cuda:0,cuda:1
```

### Memory Management

```bash theme={null}
# Force model to stay in RAM
./llama-server -m model.gguf --mlock

# Disable memory mapping
./llama-server -m model.gguf --no-mmap

# Set cache RAM limit (MiB, -1 = unlimited)
./llama-server -m model.gguf -cram 8192
```

### KV Cache Optimization

```bash theme={null}
# Quantize KV cache
./llama-server -m model.gguf -ctk q8_0 -ctv q8_0

# Use unified KV buffer
./llama-server -m model.gguf -kvu

# Disable KV offload
./llama-server -m model.gguf -nkvo
```

## Logging

```bash theme={null}
# Set verbosity level (0-4)
./llama-server -m model.gguf -lv 3

# Enable verbose logging
./llama-server -m model.gguf -v

# Log to file
export LLAMA_LOG_FILE="server.log"
./llama-server -m model.gguf

# Enable timestamps
./llama-server -m model.gguf --log-timestamps

# Colored logs (auto/on/off)
./llama-server -m model.gguf --log-colors auto
```

## Timeout Configuration

```bash theme={null}
# Set read/write timeout (seconds)
./llama-server -m model.gguf --timeout 600

# Time limit for text generation (milliseconds)
curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Generate",
    "t_max_predict_ms": 5000
  }'
```

## Building from Source

```bash theme={null}
# Build with CMake
cmake -B build
cmake --build build --config Release -t llama-server

# Binary location
./build/bin/llama-server
```

### With SSL Support

```bash theme={null}
cmake -B build -DLLAMA_OPENSSL=ON
cmake --build build --config Release -t llama-server
```

## Common Configurations

### High-Throughput Server

```bash theme={null}
./llama-server -m model.gguf \
  -c 4096 \
  -np 8 \
  -b 2048 \
  -ub 512 \
  -ngl 99 \
  -t 16 \
  --cache-prompt \
  --cache-reuse 256 \
  --host 0.0.0.0 \
  --port 8080
```

### Low-Latency Server

```bash theme={null}
./llama-server -m model.gguf \
  -c 2048 \
  -np 2 \
  -b 512 \
  -ub 256 \
  -ngl 99 \
  -fa on \
  --cache-prompt \
  --host 0.0.0.0 \
  --port 8080
```

### Development Server

```bash theme={null}
./llama-server -m model.gguf \
  -c 2048 \
  -ngl 99 \
  -v \
  --metrics \
  --props \
  --log-timestamps \
  --host 127.0.0.1 \
  --port 8080
```

## See Also

<CardGroup cols={2}>
  <Card title="CLI Tool" icon="terminal" href="/inference/cli">
    Command-line inference interface
  </Card>

  <Card title="Embeddings" icon="diagram-project" href="/inference/embeddings">
    Generate text embeddings
  </Card>

  <Card title="Multimodal" icon="image" href="/inference/multimodal">
    Vision and audio support
  </Card>

  <Card title="Speculative Decoding" icon="forward-fast" href="/inference/speculative-decoding">
    Accelerate with draft models
  </Card>
</CardGroup>