Run a vLLM inference server
A small-model smoke test with local-only access, health checks and cleanup.
1. Prepare a compatible host
Complete GPU setup first. Select a released vLLM image tag compatible with the GPU architecture and driver, and set ARVICA_VLLM_IMAGE. Record its digest for repeatability. This small-model example is a connectivity check, not a production benchmark or a preinstalled Arvica template.
2. Start and check
Run on the GPU host. Ports are bound to loopback. Wait for model loading before health checks. /health should return HTTP 200 and /v1/models should list the selected model. Model downloads require outbound network access and enough disk space.
#!/usr/bin/env bash
set -euo pipefail
: "${ARVICA_VLLM_IMAGE:?Set a tested vllm/vllm-openai image tag or digest}"
nvidia-smi >/dev/null
docker run -d --name arvica-vllm --gpus all --shm-size=8g \
-p 127.0.0.1:8000:8000 -v arvica-hf-cache:/root/.cache/huggingface \
"$ARVICA_VLLM_IMAGE" --model Qwen/Qwen2.5-0.5B-Instruct \
--max-model-len 2048 --gpu-memory-utilization 0.8
# Model download and warm-up can take several minutes.
docker logs --tail 80 arvica-vllm
curl --fail http://127.0.0.1:8000/health
curl --fail http://127.0.0.1:8000/v1/models
3. Connect through SSH
On your computer, keep this SSH session open; then call localhost:8000 from your client. For public production access, add authentication and TLS rather than exposing this test port.
ssh -i ~/.ssh/arvica_ed25519 -N -L 8000:127.0.0.1:8000 ubuntu@INSTANCE_IP4. Diagnose and stop
If out of memory, reduce context length or concurrency and check other GPU processes. For architecture errors, select a compatible image. Connection refused during warm-up: inspect logs. Stop only this example container with docker stop arvica-vllm; remove it with docker rm arvica-vllm. The model cache volume remains. Stopping a container does not stop cloud-instance billing.