vLLM for open source LLMs on HPC

Introduction

vLLM is an open-source library for running large language models (LLMs) quickly and efficiently on GPUs. You load a trained model and serve it so applications can send requests and get responses.
Alongside engines such as SGLang and TensorRT-LLM, vLLM is a common choice for high-throughput inference on CUDA hardware.
This post will create a conda env, install vLLM, serve Hugging Face models, and chat from Python with an OpenAI-compatible client.

What does the “v” in vLLM stand for?

Originally, “v” stood for virtual, reflecting the goal of making large models lighter to deploy.

Create a conda environment and install vLLM

On HPC, load the modules you need, create an environment, and install vLLM with a CUDA-capable PyTorch backend:

$ module load conda gcc/13 cuda/12
$ conda create --prefix=~/vllm python=3.12
$ conda activate ~/vllm
$ uv pip install vllm --torch-backend=auto

Request a compute node with at least one GPU before serving a model.

Deploy vLLM

Download and serve a model

vLLM downloads models from Hugging Face. Examples used in the notebook:
- openai/gpt-oss-20b
- google/gemma-4-E2B

$ vllm serve openai/gpt-oss-20b
$ vllm serve google/gemma-4-E2B

Where models are stored

By default, Hugging Face (and therefore vLLM) caches models under your home directory in ~/.cache:

$ ls ~/.cache/huggingface/hub | grep models--

On regular HPC, home storage is limited. Prefer project storage by setting HF_HOME, then download again and list models there:

$ export HF_HOME=/project_storage
$ ls $HF_HOME/hub

Serving and ports

Serve the model on the GPU node where your job is running:

$ vllm serve openai/gpt-oss-20b

By default the server listens on http://0.0.0.0:8000.
Shared nodes often have port conflicts. Pick your own port:

$ vllm serve openai/gpt-oss-20b --port 1234

Run chat from Python or Jupyter

Once the server is up, talk to it with the OpenAI Python client pointed at the local vLLM endpoint (API key can be any placeholder):

from openai import OpenAI

question = "write a poem about SMU in Dallas"

client = OpenAI(
    base_url="http://localhost:1234/v1",
    api_key="EMPTY",  # vLLM usually accepts any placeholder
)

resp = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": question}],
    temperature=0,
)

print(resp.choices[0].message.content)

Install the client in the same conda env if needed: pip install openai.

Serving multiple models

You may want several LLMs at once—for example one for text chat, one for vision, one for code.
Host each model on a different port and cap GPU memory per process with --gpu-memory-utilization so they do not overrun the GPU:

$ vllm serve openai/gpt-oss-20b --port=1234 --gpu-memory-utilization 0.30 &
$ vllm serve google/gemma-4-E2B-it --port=4321 --gpu-memory-utilization 0.30 &

Call each server from Python with its own client and base_url:

from openai import OpenAI

question = "write a poem about SMU in Dallas"

client1 = OpenAI(
    base_url="http://localhost:1234/v1",
    api_key="EMPTY",
)
resp1 = client1.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": question}],
    temperature=0,
)
print(resp1.choices[0].message.content)

client2 = OpenAI(
    base_url="http://localhost:4321/v1",
    api_key="EMPTY",
)
resp2 = client2.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[{"role": "user", "content": question}],
    temperature=0,
)
print(resp2.choices[0].message.content)

Example output from the Gemma server in the notebook:

## The Spirit of SMU in Dallas

Where the DFW winds begin to blow,
And the city's vibrant currents flow,
Stands a beacon, strong and bright,
...

Stop vLLM

To stop all vllm serve processes on the node:

$ pkill -f "vllm serve"

Summary

Create a conda env → install vLLM with CUDA → set HF_HOME if needed → vllm serve on a free port → chat via the OpenAI-compatible API.
Full notebook: Vllm.ipynb.
Official docs: vLLM documentation.