Instead of stopping a model when llama-swap swaps it out, llama-sleep moves the model's GPU memory into host RAM with the CUDA driver's checkpoint API and keeps the process alive. The next swap-in copies it back: no model load, and the KV cache, including cached prompts, is still there.

It follows the pattern of llama-swap's vllm-wrapper: llama-swap stays unchanged, and llama-sleep is used as a model's cmd and cmdStop.

Results

Measured on an RTX 3060 12 GB (PCIe 3.0 x16) with llama-swap and llama.cpp b11552, two Gemma 4 models that each fill the GPU. Times are from the request until the first token.

	Stop and cold start	llama-sleep	llama-sleep + mTHP
Swap between two 26B models	10.3 s	2.1 s	1.7 s
Swap from the 26B to the 12B	5.1 s	1.9 s	1.1 s
A 9,589-token prompt sent again after a swap	9,589 tokens recomputed	1 token recomputed	1 token recomputed

In a swap with llama-sleep, nearly all of the time is waking the model: copying its GPU memory back from RAM. Everything else takes about 80 ms. "mTHP" is the kernel's multi-size transparent huge pages, which make both copy directions about twice as fast; see docs/cuda.md.

How it works

                 llama-swap
                     │ ${PORT}
          ┌──────────▼──────────┐
          │  llama-sleep serve  │  reverse proxy; holds every request
          └──────────┬──────────┘  until the model is awake and healthy
                     │ Unix socket or container port
          ┌──────────▼──────────┐
          │    llama-server     │  keeps running while asleep
          └─────────────────────┘
       awake: weights + KV cache in GPU memory
       asleep: the same bytes in host RAM, GPU free

serve (llama-swap's cmd) starts llama-server as a detached daemon, or reuses a sleeping one with the same command line, and wakes it right away. It proxies llama-swap's port to it and holds every request, llama-swap's health check included, until the model can work.
sleep (llama-swap's cmdStop) puts the daemon to sleep, and only then ends serve, so llama-swap starts the next model on a free GPU.
--idle puts a model to sleep in the background after it has been idle for that long. The next swap then only has to wake the new model, and the slower copy direction (GPU to RAM) happens while nobody waits.

Details: docs/behaviour.md (life cycle, modes, failure handling) and docs/cuda.md (the CUDA checkpoint API and what to know about it).

Requirements

Linux on x86_64, an NVIDIA GPU and driver 570 or newer (the driver's checkpoint API in libcuda.so.1). No CUDA toolkit and no cuda-checkpoint.
llama.cpp's llama-server, started either directly or with podman run or docker run.
Enough host RAM for every sleeping model: about its GPU memory use.
Recommended: multi-size transparent huge pages (Linux 6.8+); llama-sleep warns if they are off.

Install

Download llama-sleep-<version>-linux-amd64 and its .sha256 from the releases page, check it and put it on the PATH:

sha256sum -c llama-sleep-<version>-linux-amd64.sha256
install -m 0755 llama-sleep-<version>-linux-amd64 ~/.local/bin/llama-sleep

Or build it (Go 1.25 or newer):

make build        # or: CGO_ENABLED=0 go build -o llama-sleep .
make check        # gofmt, go vet, tests, build

The result is a single binary without cgo. It loads libcuda.so.1 at run time (and libnvidia-ml.so.1, if present, for the memory check).

llama-sleep check then tells whether the system can put models to sleep and what slows it down:

ok    CUDA driver API 13.0 with the checkpoint API
ok    GPU device accessible
ok    NVIDIA driver 580.95.05 (NVML: memory checks before sleeping and waking)
warn  multi-size transparent huge pages are off (...); sleeping and waking are about twice as slow
ok    70.2 GiB of host memory available for sleeping models
ok    state directory /run/user/1000/llama-sleep

Use with llama-swap

llama-server in containers

Each model runs as its own container. The container publishes LLAMA_SLEEP_PORT, which llama-sleep replaces with llama-swap's port + 1000; --name is required.

macros:
  "serve": llama-sleep serve --name ${MODEL_ID} --listen 127.0.0.1:${PORT} --idle 10s --
  "sleep": llama-sleep sleep --name ${MODEL_ID} --stop-pid ${PID}

models:
  "gemma-26b":
    cmd: >-
      ${serve} podman run --rm --init --name llm-gemma-26b
      --device nvidia.com/gpu=all -p 127.0.0.1:LLAMA_SLEEP_PORT:8080
      ghcr.io/ggml-org/llama.cpp:server-cuda
      --host 0.0.0.0 --port 8080 -hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL
    cmdStop: ${sleep}

llama-server as a process

llama-sleep appends --host <state-dir>/<name>.sock, so llama-server listens on a Unix socket that only llama-sleep uses.

models:
  "gemma-26b":
    cmd: >-
      llama-sleep serve --name ${MODEL_ID} --listen 127.0.0.1:${PORT} --idle 10s --
      llama-server -hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL
    cmdStop: llama-sleep sleep --name ${MODEL_ID} --stop-pid ${PID}

Both examples are complete in examples/.

Options

llama-sleep serve [options] -- <command>

Option	Default	Meaning
`--name`		Unique per model; use `${MODEL_ID}`
`--listen`		Address llama-swap proxies to: `127.0.0.1:${PORT}`
`--idle`	`0` (off)	Put the model to sleep after this much idle time
`--ready-timeout`	`10m`	How long to wait for llama-server to become healthy
`--port-offset`	`1000`	Container mode: `LLAMA_SLEEP_PORT` = listen port + offset
`--ttl`	`0` (never)	End the model once it has slept this long, freeing its host memory
`--state-dir`	`$XDG_RUNTIME_DIR/llama-sleep`	Sockets, pid files, logs

llama-sleep sleep [options]

Option	Default	Meaning
`--name`		As given to `serve`
`--stop-pid`		The `serve` process to end afterwards: `${PID}`
`--mode`	`checkpoint`	`checkpoint`: put to sleep; `stop`: end llama-server unless it already sleeps
`--state-dir`	`$XDG_RUNTIME_DIR/llama-sleep`	As given to `serve`

Other commands:

Command	Does
`llama-sleep status`	Each model: awake or asleep, since when, host memory held, TTL, last sleep and wake time
`llama-sleep status --prometheus`	The same as Prometheus metrics, plus counters (cold starts, sleeps, wakes, stops, failures), e.g. for node_exporter's textfile collector
`llama-sleep check`	Driver, GPU access, huge pages, memory, state directory
`llama-sleep gc`	End models that slept longer than their `--ttl`; a running `serve` does this every 30 s, `gc` is for a timer

Choosing `--idle` and `--mode`

A model that is swapped away often and used in bursts (batch jobs, automations): --idle of a few seconds. Swaps away from it then cost almost nothing.
A model that must answer instantly after a pause (a voice assistant): no --idle. It is put to sleep only when a swap needs the GPU, so a request after a quiet minute does not have to wake it first.
--mode stop frees the GPU fastest when a model is swapped away while awake, but loses the KV cache and cold-starts next time.

Limitations

Every sleeping model keeps its GPU memory in host RAM.
A swap that arrives while a model is still being put to sleep waits for it (the copy to RAM takes longer than the copy back).
NVIDIA and Linux only; tested with one GPU per process.
A sleeping llama-server must be ended with SIGKILL; llama-sleep does that itself, but keep it in mind for scripts that stop its containers.

License

MIT, see LICENSE.

README.md

llama-sleep