How to Set Up a Private Cloud GPU Server with Ollama? GPU Passthrough in 6 Steps

The complete path to set up an Ollama private cloud GPU server is exactly 6 steps: install NVIDIA Container Toolkit, use docker run to pass through GPU, write a Compose file to solidify configuration, mount persistent volumes to save weights, isolate by GPU, and open an OpenAI-compatible interface—no distributed cluster needed. By August 2026, this path has been standardized in private deployment practices, covering both single-GPU and multi-GPU setups.

Confirm Three Things Before Setting Up Ollama Private Cloud GPU Server: Driver, GPU Count, Model Scale

First, take two minutes to confirm three things:

• Is nvidia-smi working: It should list GPUs, driver, and VRAM.
• How many GPUs: nvidia-smi -L to see GPU IDs, decide whether to use count: all or device_ids.
• Maximum model size: 7B or 70B quantized, determines VRAM requirements and Compose file.

Step 1: Install NVIDIA Container Toolkit

To use GPUs in containers, you must have NVIDIA Container Toolkit as a bridge. If you only installed the driver without the toolkit, or didn't restart Docker after installation, docker run --gpus all will report:

could not select device driver "" with capabilities: [[gpu]]

Troubleshooting order: confirm the toolkit is installed (dpkg -l | grep nvidia-container-toolkit), restart Docker (sudo systemctl restart docker), verify (docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi).

Step 2: Start a Minimal Service with docker run

First, run a one-off command to verify GPU passthrough:

docker run -d --gpus all -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

• --gpus all: Allows the container to access all GPUs.
• -v ollama:/root/.ollama: Persists model weights.
• -p 11434:11434: Exposes the default port.

Enter the container and run nvidia-smi to confirm GPUs are visible, then pull the model.

Step 3: Write Docker Compose to Solidify Configuration

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  ollama:

• deploy.resources.reservations.devices: Declare GPU resource reservations.
• count: all: Expose all GPUs.
• restart: unless-stopped: Automatically restart on crash.

To specify a particular GPU, use device_ids: ["0", "1"], but note that device numbering inside the container is remapped starting from 0.

Step 4: Mount Persistent Volumes to Save Model Weights

By default, Ollama models are stored in /root/.ollama inside the container. Without mounting a volume, weights are lost when the container is rebuilt.

For first-time deployment, consider renting instances on-demand to test before deciding on long-term retention.

Step 5: Multi-GPU Isolation—Use NVIDIA_VISIBLE_DEVICES to Assign GPUs

On multi-GPU machines, two scenarios:

• Scenario A: Run two service instances, each using one GPU.
• Scenario B: Reserve some GPUs for other tasks.

GPU numbering inside the container is remapped from 0; don't hardcode the wrong numbers in scripts.

Step 6: Expose OpenAI-Compatible API and Protect Access

Ollama natively provides an OpenAI-compatible API. In container deployment, the -p port mapping determines reachability; test with curl http://服务器IP:11434/v1/models. However, there is no authentication by default; three safe exposure methods:

1. Reverse proxy with authentication: Use Nginx or Caddy for TLS and API-KEY.
2. Security group whitelist: Only allow specific IPs.
3. Internal network or tunnel: VPN or intranet penetration.

VRAM Scheduling: Allocating When Multiple Models Stay Resident

When multiple models are resident, VRAM is occupied by weights, KV cache, and concurrent increments. For allocation algorithms, refer to How to Choose GPU VRAM. If VRAM is insufficient when loading multiple models simultaneously, it's often due to idle residency time being too long.

Below are estimated ranges based on quantization precision; actual values vary with context length and concurrency:

If VRAM is insufficient, use OLLAMA_KEEP_ALIVE=0 to release memory immediately after use.

GPU Model Comparison: VRAM Steps for 7B/14B/32B/70B Quantized Versions

Based on August 2026 cloud compute data, L40S has 48GB, rental about $1.50/GPU-hour; A100 80GB rental about $1.77/GPU-hour. VRAM estimates (based on quantization precision, actual values vary):

For batch ≥ 8 with FP8, L40S offers better cost; for ultra-long context and low batch, A100 has lower latency. It's recommended to test on a platform that supports on-demand, multiple GPU models, and pre-built templates (like NexGPU) before deciding on long-term specs. See L40S Rental and How Much VRAM Does Qwen Deployment Need.

Setup Checklist and Common Pitfalls

• [ ] NVIDIA Container Toolkit installed, Docker restarted
• [ ] After docker run --gpus all, nvidia-smi shows GPUs in container
• [ ] Weight volume mounted (/root/.ollama persisted)
• [ ] GPU-level isolation configured
• [ ] External port protected with authentication

FAQ

How to specify which GPU to use in Ollama container?

Use environment variable NVIDIA_VISIBLE_DEVICES, e.g., -e NVIDIA_VISIBLE_DEVICES=1, or write device_ids: ["1"] in Compose. Inside the container, numbering starts from 0.

docker run ollama reports 'could not select device driver nvidia'?

Check if the toolkit is installed, restart Docker; if still error, run nvidia-smi to confirm driver.

Where are Ollama model files stored in the container?

By default in /root/.ollama; use -v ollama:/root/.ollama or bind to a host directory for persistence.

What if Ollama loads multiple models and runs out of VRAM?

Set OLLAMA_KEEP_ALIVE=0 to release memory, or switch to smaller quantizations, reduce concurrency.

How much VRAM is needed for running 70B quantized model on Ollama?

Estimated ~35-45 GB (Q4_K_M), varies with context and concurrency. Recommend renting 48GB or 80GB on-demand to test.

First, use NexGPU on-demand instances to run through the six steps, measure actual VRAM consumption, then decide on long-term specs.