> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/ggml-org/llama.cpp/llms.txt
> Use this file to discover all available pages before exploring further.

# Introduction

> LLM inference in C/C++ with minimal setup and state-of-the-art performance

# Welcome to llama.cpp

llama.cpp enables **LLM inference with minimal setup** and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

<CardGroup cols={2}>
  <Card title="Get Started" icon="rocket" href="/quickstart">
    Run your first LLM in minutes
  </Card>

  <Card title="API Reference" icon="code" href="/api/libllama">
    Explore the C/C++ API
  </Card>

  <Card title="Model Support" icon="brain" href="/models/supported-models">
    Browse 100+ supported models
  </Card>

  <Card title="Build Guide" icon="hammer" href="/development/building">
    Compile from source
  </Card>
</CardGroup>

## Why llama.cpp?

<CardGroup cols={2}>
  <Card title="Minimal Dependencies" icon="circle-nodes">
    Plain C/C++ implementation with no external dependencies
  </Card>

  <Card title="Multi-Platform" icon="globe">
    Runs on CPU, GPU, and NPU across x86, ARM, and mobile platforms
  </Card>

  <Card title="Quantization" icon="compress">
    1.5-bit to 8-bit integer quantization for reduced memory and faster inference
  </Card>

  <Card title="Hardware Acceleration" icon="microchip">
    Optimized kernels for Metal, CUDA, Vulkan, SYCL, and more
  </Card>
</CardGroup>

## Key Features

* **Apple Silicon First-Class Support** — Optimized via ARM NEON, Accelerate, and Metal frameworks
* **x86 Acceleration** — AVX, AVX2, AVX512, and AMX support
* **GPU Support** — Custom CUDA kernels for NVIDIA, HIP for AMD, MUSA for Moore Threads
* **OpenAI-Compatible Server** — Drop-in replacement with `/v1/chat/completions` endpoint
* **Multimodal Models** — Support for vision models like LLaVA and Qwen2-VL
* **Speculative Decoding** — Parallel decoding for improved throughput
* **CPU+GPU Hybrid Inference** — Partially accelerate models larger than VRAM capacity

## Quick Example

<CodeGroup>
  ```bash CLI theme={null}
  # Download and run a model from Hugging Face
  llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

  # Or use a local GGUF file
  llama-cli -m my_model.gguf -p "Hello, world!"
  ```

  ```bash Server theme={null}
  # Launch OpenAI-compatible API server
  llama-server -hf ggml-org/gemma-3-1b-it-GGUF

  # Server runs on http://localhost:8080
  # API endpoint: http://localhost:8080/v1/chat/completions
  ```

  ```c C API theme={null}
  #include "llama.h"

  // Initialize model
  llama_model_params model_params = llama_model_default_params();
  llama_model * model = llama_model_load_from_file("model.gguf", model_params);

  // Create context
  llama_context_params ctx_params = llama_context_default_params();
  llama_context * ctx = llama_context_new_with_model(model, ctx_params);

  // Tokenize and decode
  std::vector<llama_token> tokens = llama_tokenize(ctx, "Hello", true, true);
  llama_decode(ctx, llama_batch_get_one(tokens.data(), tokens.size()));
  ```
</CodeGroup>

## Supported Models

llama.cpp supports 100+ model architectures including:

* **LLaMA** — LLaMA, LLaMA 2, LLaMA 3, and fine-tunes
* **Mistral** — Mistral 7B, Mixtral MoE, and variants
* **Gemma** — Google Gemma and Gemma 2 models
* **Qwen** — Qwen and Qwen2 series
* **Phi** — Microsoft Phi models and PhiMoE
* **Vision Models** — LLaVA, Qwen2-VL, MiniCPM, and more

[View full model support →](/models/supported-models)

## Performance

llama.cpp is optimized for inference across diverse hardware:

| Backend | Target Platform             |
| ------- | --------------------------- |
| Metal   | Apple Silicon (M1/M2/M3/M4) |
| CUDA    | NVIDIA GPUs                 |
| HIP     | AMD GPUs                    |
| Vulkan  | Cross-platform GPU          |
| SYCL    | Intel GPU and Nvidia GPU    |
| CANN    | Ascend NPU                  |
| OpenCL  | Adreno GPU                  |

## Community & Ecosystem

llama.cpp powers a vibrant ecosystem of tools, UIs, and language bindings:

* **Language Bindings** — Python, Node.js, Rust, Go, C#, Java, Swift, and more
* **UIs** — LM Studio, ollama, text-generation-webui, Jan, and dozens more
* **Tools** — Model conversion, quantization, benchmarking, and evaluation

## Next Steps

<CardGroup cols={2}>
  <Card title="Quickstart Guide" icon="play" href="/quickstart">
    Run your first inference in 5 minutes
  </Card>

  <Card title="Installation" icon="download" href="/installation">
    Install via package manager or build from source
  </Card>

  <Card title="Core Concepts" icon="book" href="/concepts/gguf-format">
    Understand GGUF format and quantization
  </Card>

  <Card title="REST API" icon="plug" href="/api/rest/overview">
    Use the OpenAI-compatible HTTP server
  </Card>
</CardGroup>