SGLang for open source LLMs on HPC

What is SGLang?

• SGLang is a high-performance serving framework for large language models and multimodal models.
• This post follows the notebook SgLang.ipynb.

Create conda env sglang and install

• Create a conda environment
• Install Vllm with Cuda enable on SuperPOD

$ module load conda gcc/13 cuda/12
$ conda create --prefix=~/sglang python=3.12
$ conda activate ~/sglang
$ uv pip install sglang sglang-kernel \
  --extra-index-url https://sgl-project.github.io/whl/cu129/ \
  --extra-index-url https://download.pytorch.org/whl/cu129 \
  --index-strategy unsafe-best-match

Deploy sglang

Download sglang model

• sglang downloads and uses Huggingface model, for example, one can download models from HuggingFace using command:
  • https://huggingface.co/openai/gpt-oss-20b
  • https://huggingface.co/google/gemma-4-E2B

$ python -m sglang.launch_server --model-path openai/gpt-oss-20b --port 30000

Key flags:

• –model-path: Hugging Face model or local path
• –port: API port
• –host: default 0.0.0.0 (accessible externally)
• –dtype: float16 or bfloat16
• –tensor-parallel-size: multi-GPU scaling

List downloaded models:

• By default, vLLM (and HuggingFace) downloads model to user’s home directory, under hidden .cache folder

$ ls ~/.cache/huggingface/hub | grep models--

• Due to limited personal home folder’s storage in SuperPOD, we encourage user to download models to project storage allocated by Cold Front HPC management system by setting HF_HOME:

$ export HF_HOME=/project_storage

• Redownload again and check the downloaded models:

$ ls $HF_HOME/hub

Run model’s chat

• Once served, you can run the chat in python code or in Jupyter Notebook:

import openai
question = "write a poem about SMU"

client = openai.Client(base_url=f"http://127.0.0.1:2346/v1", api_key="None")

response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[
        {"role": "user", "content": question},
    ],
    temperature=0,
    max_tokens=64,
)
print(response.choices[0].message.content)

Serving multiple vLLMs

• Sometime user want to run 2 or more LLMs, just like in a chatbot where 1 LLM is for regular answer, another is for image querying and another for code.
• We can host 2 different models on 2 different port numbers with pre-allocated gpu-memory-utilization for each model size to avoid cuda overhead

$ python -m sglang.launch_server --model-path openai/gpt-oss-20b --port=1234 & 

$ python -m sglang.launch_server --model-path google/gemma-4-E2B-it --port=4321 & 

from openai import OpenAI

question = "write a poem about SMU in Dallas"

client1 = OpenAI(
    base_url="http://127.0.0.1:5436/v1",
    api_key="EMPTY",  # vLLM usually accepts any placeholder
)

resp1 = client1.chat.completions.create(
    model="google/gemma-3n-E4B-it",
    messages=[{"role": "user", "content": question}],
    temperature=0,
)

print(resp1.choices[0].message.content)

To kill vllm instance:

$ pkill -f sglang