Original date published: June 3, 2026

Building KAARL - A multi-node AI cluster infrastructure for rapid prototyping

Building KAARL - A multi-node AI cluster infrastructure with 4 x 4060 Ti, CUDA, Kubernetes and Ray to power AI workloads and experiments

Weights of Frustration & Sparks of Curiosity

While working on a project to train a small transformer model from the Andrej Karpathy NanoGPT tutorial, I got frustrated with the constant dependency on cloud providers for GPUs. You are either constantly battling for availability or having to perform tricks to prevent your current GPU allocation from being reallocated. If you've worked on AI-heavy projects, you'd know that in the very early experimentation phases, you need stable GPU availability to run multiple different experiments. After jumping across several subscriptions from various cloud providers, I was sufficiently pissed off. I had also been reading a lot about GPUs, infrastructure, home servers, local AI, etc.

As any sane person would conclude, the next obvious step is to build my own GPU infrastructure. I needed something that was good enough for me to run small experiments locally that I could then move to larger instances when I was sure of the nature of the experiment or wanted to run bigger model variants. The platform would serve as a testbed for my curiosities.

So, I went to Amazon and started picking out the components I would need. I settled on 4 x 4060 Ti GPUs (16GB VRAM) distributed across two nodes, with each node having 2 GPUs. Each node also has 2 x 32GB of memory and 512GB of SSD storage attached. I also included an 8TB HDD storage disk attached to one of the nodes to serve as a network file system; this would be used to store the various model weights and datasets I would be using for my experiments. I also chipped in 4 x NVIDIA Jetson Nanos for running edge robotics projects.

There are a lot more materials and components used for building this. You can find the BOM here. You can use this as a reference for your own build as well. The rest of this post goes over the details of how I built the infrastructure and set up CUDA, Kubernetes and Ray to run AI projects.

The hardware

Component list

The cluster primarily consists of two identical nodes. Each node was built from individual components.

Total across cluster:

• 4× RTX 4060 Ti (64GB VRAM total)
• 16 CPU cores per node (32 total)
• 128GB RAM total
• ~6.5TB usable storage

GPU Purchase Notes

The 4× RTX 4060 Ti 16GB cards were purchased separately from Amazon. Key considerations when buying:

• Buy the 16GB variant — the 8GB version exists and is cheaper, but 16GB is essential for running large models. Double-check the listing.
• Check PCIe slot availability — the MSI B550 Tomahawk has 2× PCIe x16 slots, accommodating 2 GPUs per board comfortably.
• PSU headroom — each RTX 4060 Ti has a 165W TDP. With CPU and other components, budget at least 650–750W per node.

Physical Assembly Notes

• The open rack chassis means no traditional case airflow. Ensure GPUs have clearance and adequate airflow across the PCIe slots.
• Seat both GPUs firmly in the PCIe x16 slots and connect the 8-pin/16-pin power connectors from the PSU.
• Connect both nodes to a dedicated managed/unmanaged network switch via Ethernet. This switch handles all cluster communication (Kubernetes, Ray, NCCL for GPU-to-GPU data transfer).

BIOS configuration

Once we have the actual hardware set up, we need to configure our BIOS (the software that loads before our operating system) to load NVIDIA drivers for our GPUs when available. This is one of those subtle steps that can easily be missed and can even add a few hours or days to the setup process. For my cluster, I'm making use of Ubuntu 24.04. The BIOS process may differ based on the motherboard you are using, but I'll go over the steps I used in my own setup.

Enter BIOS

Power on the machine and press the Delete button repeatedly to enter the MSI BIOS page. Disable Secure Boot (required for NVIDIA driver installation):

Settings → Security → Secure Boot → Disabled

If Secure Boot is left enabled, the NVIDIA kernel module will fail to load after driver installation. You will see a black screen or nvidia-smi will return No devices found.

BIOS changes

These are not strictly required but improve stability:

• Enable Above 4G Decoding — allows the system to properly address GPU memory above 4GB over PCIe

  Settings → Advanced → PCI Subsystem Settings → Above 4G Memory / Crypto Currency Mining → Enabled
  

• Enable Virtualization (AMD-V / SVM) — required for some container runtimes and future VM workloads

  Settings → OC → CPU Features → SVM Mode → Enabled
  

Save and exit. The machine will reboot.

Choosing and installing the right operating system

I knew from the get-go I would have to use Linux for the cluster, but then I had to pick the right distro and the right version. I settled on Ubuntu 24.04 due to its recency and support for most deep learning libraries, CUDA and ROS.

Installation process

On another machine, download Ubuntu 24.04 LTS and flash it to a USB drive:

# Download Ubuntu 24.04 LTS
# https://ubuntu.com/download/server

# Flash to USB (replace /dev/sdX with your USB device)
sudo dd if=ubuntu-24.04-live-server-amd64.iso of=/dev/sdX bs=4M status=progress
sync

Alternatively, use Balena Etcher on Windows or macOS. Boot from the USB drive (press F11 on MSI boards for boot menu) and follow the installer:

• Language: English
• Keyboard: Your layout
• Installation type: Ubuntu Server (no GUI needed on worker nodes; install desktop on Node 1 if you want a local display)
• Storage: Use the 512GB SSD as the installation target. Leave the 8TB HDD unconfigured for now (Node 1 only).
• Network: Configure Ethernet. Note the IP addresses assigned to each node.
• Profile setup:
  • Node 1: username kaarl, hostname kaarl-bignode1
  • Node 2: username kaarl, hostname kaarl-bignode2
• SSH: Enable OpenSSH server during installation

On both nodes:

sudo apt update && sudo apt upgrade -y
# Optionally install the desktop GUI if you are more comformatable with that
sudo apt install ubuntu-desktop
# Restart the server
sudo reboot

Network Configuration

Kubernetes and Ray require stable node addresses. Configure static IPs via Netplan:

sudo nano /etc/netplan/00-installer-config.yaml

network:
  version: 2
  ethernets:
    enp3s0:  # Replace with your interface name (check with: ip link show)
      dhcp4: no
      addresses:
        - 192.168.1.101/24  # Node 1
        # - 192.168.1.102/24  # Node 2 (on the other machine)
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]

sudo netplan apply

Verify:

ip addr show
ping kaarl-bignode2  # Ping the other node

To configure /etc/hosts, add entries on both nodes so hostnames resolve without DNS:

sudo nano /etc/hosts

192.168.1.101  kaarl-bignode1
192.168.1.102  kaarl-bignode2

Now let's configure SSH key access between nodes:

# On Node 1: generate key pair
ssh-keygen -t ed25519 -C "kaarl-cluster"

# Copy public key to Node 2
ssh-copy-id kaarl@kaarl-bignode2

# Test
ssh kaarl@kaarl-bignode2

Setting up NVIDIA drivers and CUDA

Now we have our compute nodes built and set up. You can start SSHing into them and running programs on them. Our use case is focused on deep learning and robotics. Therefore, we need an accelerator for performing scientific computing operations, and this is where NVIDIA and CUDA come in. NVIDIA provides the accelerator hardware, which can be used for graphics by default, but we want to write and run scientific computing applications on it, and to do so we will need to install CUDA. CUDA is the library that actually allows us to utilize the vector computation built natively into the GPU to run our accelerated computing workloads. Most of the libraries we would be using for our experiments, such as PyTorch, have already written the code that bridges our application use case to CUDA, so in most cases we would not need to write any CUDA code ourselves. We are setting up CUDA for our AI libraries to use. If you've seen the memes online, you'd know that this is one of the more annoying steps. You'll want to be careful here and verify your work at each step.

CUDA installation process

Repeat on both nodes.

Verify GPUs are detected

lspci | grep -i nvidia
# Should show 2 entries per node:
# VGA compatible controller: NVIDIA Corporation ...
# Audio device: NVIDIA Corporation ...

Install NVIDIA drivers via apt

# Check which driver versions are available
apt-cache search nvidia-driver

# Install the recommended driver (560 series for RTX 4060 Ti on Ubuntu 24.04)
sudo apt install nvidia-driver-560 -y

# Reboot is required after driver installation
sudo reboot

Verify Driver Installation

nvidia-smi

Expected output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05    Driver Version: 560.35.05    CUDA Version: 12.6               |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
|   0  NVIDIA GeForce RTX 4060 Ti     Off | 00000000:04:00.0  ...  |                  N/A |
|   1  NVIDIA GeForce RTX 4060 Ti     Off | 00000000:2B:00.0  ...  |                  N/A |
+-----------------------------------------------------------------------------------------+

If nvidia-smi fails:

• Confirm Secure Boot is disabled in BIOS
• Check dmesg | grep -i nvidia for errors
• Ensure the driver version matches your kernel: apt install nvidia-dkms-560

Install CUDA Toolkit

The NVIDIA driver already includes CUDA runtime libraries. The toolkit adds nvcc and development headers for building custom CUDA code:

# Install CUDA toolkit matching driver (12.x for driver 560)
sudo apt install nvidia-cuda-toolkit -y

# Verify
nvcc --version

We now need to persist the GPU settings. Enable persistence mode so the driver stays loaded between reboots:

sudo nvidia-smi -pm 1

# Verify
nvidia-smi | grep Persistence
# Should show: Persistence-M: On

Installing Miniconda for environment management

We will be running multiple projects in parallel with conflicting dependencies, or we could simply want one project to have a different Python version than the others. Miniconda allows us to easily configure and maintain multiple Python environments without having them tied to any directory (as done with python-venv). Miniconda environments are global and can be used from anywhere in the system, and each environment can be used by multiple projects irrespective of the folder location of each project. You do not need to track any venv or .venv folder in your project codebase because Miniconda handles the location and usage of your environments. It is very convenient if you plan to reuse certain environments across projects or if you need specific environment specifications for your project without affecting the base Python installation. The installation process is as follows.

Repeat on both nodes (or at minimum on Node 1, which you'll use as the control machine).

# Download Miniconda installer
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

# Run installer
bash Miniconda3-latest-Linux-x86_64.sh
# Follow prompts, accept license, choose install location (~miniconda3)
# When asked to initialize conda: yes

# Reload shell
source ~/.bashrc

# Verify
conda --version


# Create new environment
conda create -n <name_of_env> python=<python_version>

# Switch to newly created environment
conda activate <name_of_env>

# Exit environment
conda deactivate

# To delete enviornment first switch to base
conda activate base

# Delete the target environment
conda remove --name <name_of_env> --all

# Ensure the environment is no longer present
conda info --envs

Install K3s (Kubernetes)

We have both nodes configured, running and communicating with each other. For a lot of simple use cases this would be enough. You could simply SSH into each node and execute whatever programs you want. You can have them communicate over the network to transfer data. Even PyTorch has a set of functions to allow training across nodes on a network. Other newer libraries like Exo allow you to run large model inference across multiple accelerated nodes in a network. For our use case we would want to run complex cluster workloads and build a developer experience where we would simply specify what capacity we need and configure the environments, and the orchestration system would provide the runtime and its requirements for us.

This was to mimic the functionality typically provided by the cloud AI infrastructure companies, but running on my own hardware in my own home. We would be using Kubernetes (through K3s, which is a lightweight version of Kubernetes that fits our exact use case) for compute orchestration. Kubernetes is what directly manages our nodes, checks their health, places jobs on them, manages volumes and persistent storage, handles capacity management and recovery, etc. We would run Ray on top of this for the actual cluster, node and job management, but we will cover Ray in later sections.

The setup process for Kubernetes is node-specific and goes as follows.

Install K3s on Node 1 (Master)

# Install K3s as master node
curl -sfL https://get.k3s.io | sh -

# Check status
sudo systemctl status k3s

# Get node token (needed to join Node 2)
sudo cat /var/lib/rancher/k3s/server/node-token
# Copy this token — you'll need it in the next step

# Get master IP
hostname -I | awk '{print $1}'
# e.g. 192.168.1.101

Join Node 2 as Worker

On Node 2:

# Replace K3S_URL and K3S_TOKEN with your values from Node 1
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.1.101:6443 K3S_TOKEN=<TOKEN_FROM_NODE1> sh -

Configure kubectl on Node 1

# Copy kubeconfig to your home directory
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/config

# Add to .bashrc so it persists
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
source ~/.bashrc

Verify Cluster

kubectl get nodes
# Expected:
# NAME             STATUS   ROLES                  AGE
# kaarl-bignode1   Ready    control-plane,master   ...
# kaarl-bignode2   Ready    <none>                 ...

Note: K3s uses containerd as its container runtime, not Docker. You cannot run docker build or docker push commands. Use pre-built images from registries instead.

Install NVIDIA GPU Operator

The GPU Operator installs and manages the NVIDIA device plugin, which exposes GPUs as schedulable Kubernetes resources.

Install Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Verify
helm version

Install the GPU Operator

# Add NVIDIA Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Create namespace
kubectl create namespace gpu-operator

# Install GPU Operator
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --set driver.enabled=false  # Driver already installed via apt — don't reinstall

Important: Pass --set driver.enabled=false because you already installed the driver manually via apt. If you let the GPU Operator install the driver, it will conflict with your existing installation.

Verify GPU Operator

# Wait for all pods to be Running (takes 2-5 minutes)
kubectl get pods -n gpu-operator -w

# Verify GPUs are visible as Kubernetes resources
kubectl describe node kaarl-bignode1 | grep -A 5 "Capacity" | grep nvidia
# Expected: nvidia.com/gpu: 2

kubectl describe node kaarl-bignode2 | grep -A 5 "Capacity" | grep nvidia
# Expected: nvidia.com/gpu: 2

Test GPU Scheduling

# Run a quick GPU test pod
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: OnFailure
  containers:
  - name: gpu-test
    image: nvidia/cuda:12.1-base-ubuntu22.04
    command: ["nvidia-smi"]
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

# Check output
kubectl logs gpu-test

# Cleanup
kubectl delete pod gpu-test

Installing the KubeRay Operator

The KubeRay Operator is the service that allows our Kubernetes environment and our Ray cluster to have bidirectional communication and pass messages and instructions between each other. It is a convenience library that allows us to quickly set up Ray correctly in a Kubernetes environment.

Save as ~/ray-cluster-values.yaml:

image:
  repository: rayproject/ray
  tag: 2.40.0-py311-gpu # this is where the cluster python enviornment is configured, you can change it here if you would like to use a different python version
  pullPolicy: IfNotPresent

head:
  rayStartParams:
    dashboard-host: '0.0.0.0'
    num-cpus: '0'  # Head node coordinates only, no compute tasks
  resources:
    limits:
      cpu: "4"
      memory: "8Gi"
    requests:
      cpu: "2"
      memory: "4Gi"

worker:
  replicas: 2
  minReplicas: 2
  maxReplicas: 4
  rayStartParams:
    num-gpus: "2"
  resources:
    limits:
      cpu: "16"
      memory: "32Gi"
      nvidia.com/gpu: "2"
    requests:
      cpu: "8"
      memory: "16Gi"
      nvidia.com/gpu: "2"
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: ray.io/node-type
              operator: In
              values:
              - worker
          topologyKey: kubernetes.io/hostname

Key decisions in this config:

• num-cpus: '0' on head — prevents compute tasks landing on the head node
• podAntiAffinity — ensures the 2 workers land on different physical nodes (1 per node, 2 GPUs each)
• replicas: 2 fixed — no auto-scaling, workers always ready for instant job start

Deploy

kubectl create namespace ray

helm install ray-cluster kuberay/ray-cluster \
  --namespace ray \
  --values ~/ray-cluster-values.yaml \
  --version 1.2.2

The rayproject/ray:2.40.0-py311-gpu image is ~5GB. First pull takes 15–20 minutes per node.

Wait for Pods

kubectl get pods -n ray -w
# Wait until all 3 show 1/1 Running:
# ray-cluster-kuberay-head-xxx          1/1   Running
# ray-cluster-kuberay-workergroup-xxx   1/1   Running  (on bignode1)
# ray-cluster-kuberay-workergroup-xxx   1/1   Running  (on bignode2)

Verify Pod Distribution

kubectl get pods -n ray -o wide
# Confirm workers are on different nodes

Verify GPU Allocation

kubectl describe node kaarl-bignode1 | grep -A 10 "Allocated resources" | grep nvidia
# Expected: nvidia.com/gpu     2     2

kubectl describe node kaarl-bignode2 | grep -A 10 "Allocated resources" | grep nvidia
# Expected: nvidia.com/gpu     2     2

Using Ray from the Host

Now that our Ray operator is running properly on Kubernetes, we would like to be able to submit Ray tasks from our local code using the Ray client library.

Create Python Environment

The Ray client Python version must match the cluster exactly (the version of Python running on your cluster must be the exact same version of Python running in your environment; if this is not configured properly, it can cause nasty, confusing errors).

# Cluster runs Python 3.11 — match it exactly
conda create -n ray-client python=3.11 -y
conda activate ray-client

pip install "ray[default]==2.40.0"

Start Port-Forward

Kubernetes services are not exposed externally by default. Use port-forward to access Ray from the host:

kubectl port-forward svc/ray-cluster-kuberay-head-svc 8265:8265 10001:10001 -n ray &

• Port 8265 → Ray Dashboard (browser)
• Port 10001 → Ray Client (Python)

Test Connection

python -c "
import ray
ray.init(address='ray://localhost:10001')
print(ray.cluster_resources())
ray.shutdown()
"
# Expected: {'GPU': 4.0, 'CPU': 32.0, 'memory': ...}

Troubleshooting Port-Forward

# If port is already in use
sudo lsof -i :10001       # Find what's holding the port
pkill -9 kubectl           # Kill all kubectl processes
# Then restart port-forward

To validate GPU access, let's ensure we can actually use our GPUs to execute jobs from Ray:

import ray

ray.init(
    address="ray://localhost:10001",
    runtime_env={"pip": ["torch==2.1.0"]}
)

@ray.remote(num_gpus=1)
def test_gpu():
    import torch
    return {
        "pytorch": torch.__version__,
        "cuda": torch.cuda.is_available(),
        "gpu": torch.cuda.get_device_name(0)
    }

results = ray.get([test_gpu.remote() for _ in range(4)])
for i, r in enumerate(results):
    print(f"GPU {i}: {r}")

ray.shutdown()

Expected output:

GPU 0: {'pytorch': '2.1.0+cu121', 'cuda': True, 'gpu': 'NVIDIA GeForce RTX 4060 Ti'}
GPU 1: {'pytorch': '2.1.0+cu121', 'cuda': True, 'gpu': 'NVIDIA GeForce RTX 4060 Ti'}
GPU 2: {'pytorch': '2.1.0+cu121', 'cuda': True, 'gpu': 'NVIDIA GeForce RTX 4060 Ti'}
GPU 3: {'pytorch': '2.1.0+cu121', 'cuda': True, 'gpu': 'NVIDIA GeForce RTX 4060 Ti'}

First run: 3–5 min (Ray installs PyTorch on workers). Subsequent runs: 30 seconds (cached).

Cluster Management Scripts

Save all scripts to ~/ray-cluster-scripts/ and run chmod +x *.sh.

ray-connect.sh

#!/bin/bash
pkill -9 kubectl 2>/dev/null
sleep 1
kubectl port-forward svc/ray-cluster-kuberay-head-svc 8265:8265 10001:10001 -n ray &
sleep 3
echo "Connected — Dashboard: http://localhost:8265"

ray-status.sh

#!/bin/bash
echo "--- Pods ---"
kubectl get pods -n ray -o wide
echo ""
echo "--- GPU Allocation ---"
kubectl describe nodes | grep -A 10 "Allocated resources" | grep -E "(nvidia|cpu|memory)"
echo ""
echo "Dashboard: http://localhost:8265"

ray-scale.sh

#!/bin/bash
# Usage: ./ray-scale.sh 0   (free GPUs)  or  ./ray-scale.sh 2   (restore)
REPLICAS=$1
WORKER=$(kubectl get deployment -n ray -l ray.io/node-type=worker -o name | head -1)
kubectl scale $WORKER -n ray --replicas=$REPLICAS
echo "Workers scaling to $REPLICAS"

ray-stop.sh

#!/bin/bash
helm uninstall ray-cluster -n ray
pkill -9 kubectl
echo "Ray cluster stopped"

ray-start.sh

#!/bin/bash
helm install ray-cluster kuberay/ray-cluster \
  --namespace ray \
  --values ~/ray-cluster-values.yaml \
  --version 1.2.2
kubectl wait --for=condition=ready pod \
  -l ray.io/cluster=ray-cluster-kuberay \
  -n ray --timeout=300s
echo "Ray cluster started"

Running a simple Hugging Face Transformers inference job on our cluster

A quick sanity check to ensure everything is working properly. We will try to run DistilBERT on a single GPU using the Hugging Face inference library.

import ray
import torch

ray.init("ray://localhost:10001")

@ray.remote(num_gpus=1, runtime_env={"pip": ["transformers", "torch"]})
def run_inference(texts: list[str]) -> list[dict]:
    from transformers import pipeline
    import torch

    device = 0 if torch.cuda.is_available() else -1
    print(f"Running on: {torch.cuda.get_device_name(0) if device == 0 else 'CPU'}")

    classifier = pipeline(
        "text-classification",
        model="distilbert-base-uncased-finetuned-sst-2-english",
        device=device
    )

    return classifier(texts)


if __name__ == "__main__":
    texts = [
        "This is a fantastic piece of work",
        "I absolutely hate rainy days",
        "The weather is okay I guess"
    ]

    ref = run_inference.remote(texts)
    results = ray.get(ref)

    for text, result in zip(texts, results):
        print(f"{text!r} => {result}")

This will create a worker based on the specified runtime environment, execute the inference task and return the results outside the job worker. This means that the local environment where the Python script is being executed does not need to have a GPU; we can simply specify in Ray the environment we would like to execute that function in, and Ray handles the rest and gives us back the results, which we can then use downstream in our application.

Running vLLM Across Both Nodes for Tensor and Pipeline Parallelism

Running vLLM directly on the host only uses the 2 GPUs local to that node. To span all 4 GPUs across both nodes, use the Ray distributed executor backend.

Install vLLM

conda activate ray-client
pip install vllm

Ensure Ray Is Connected and Visible

cd ~/ray-cluster-scripts && ./ray-connect.sh

# Must show GPU: 4.0 before proceeding
python -c "import ray; ray.init(address='ray://localhost:10001'); print(ray.cluster_resources()); ray.shutdown()"

Launch vLLM Server (All 4 GPUs)

Run once on Node 1. Ray handles distribution to both nodes automatically:

vllm serve mistralai/Mistral-7B-v0.1 \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 4 \
    --distributed-executor-backend ray \
    --gpu-memory-utilization 0.90

• --distributed-executor-backend ray — connects vLLM to the Ray cluster
• --tensor-parallel-size 4 — splits the model across all 4 GPUs
• Ray places vLLM workers on both nodes automatically

For Large Models: Tensor + Pipeline Parallelism

# 2 GPUs TP per node × 2 nodes PP = 4 GPUs total
vllm serve meta-llama/Llama-2-30b-hf \
    --tensor-parallel-size 2 \
    --pipeline-parallel-size 2 \
    --distributed-executor-backend ray \
    --gpu-memory-utilization 0.95

Test the Server

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "mistralai/Mistral-7B-v0.1", "prompt": "Explain distributed computing:", "max_tokens": 100}'

Helper Script: start-vllm-ray.sh

#!/bin/bash
MODEL=${1:-"mistralai/Mistral-7B-v0.1"}
PORT=${2:-8000}
TP=${3:-4}

source ~/miniconda3/bin/activate ray-client

if ! pgrep -f "port-forward.*ray" > /dev/null; then
    echo "Ray port-forward not running. Run ./ray-connect.sh first."
    exit 1
fi

vllm serve "$MODEL" \
    --host 0.0.0.0 --port $PORT \
    --tensor-parallel-size $TP \
    --distributed-executor-backend ray \
    --gpu-memory-utilization 0.90 \
    --max-model-len 4096

Running SGLang Across Both Nodes

SGLang is an alternative LLM serving framework with faster structured generation. Setup is nearly identical to vLLM.

Install SGLang

conda activate ray-client
pip install "sglang[all]"

Launch SGLang Server (All 4 GPUs)

python -m sglang.launch_server \
    --model-path mistralai/Mistral-7B-v0.1 \
    --host 0.0.0.0 \
    --port 8000 \
    --tp 4 \
    --distributed-executor-backend ray \
    --mem-fraction-static 0.85

Key Differences from vLLM

Model Size Reference

Key Design Decisions

No auto-scaling: Kubernetes GPU allocation does not block host process access to GPUs. Host tools like Ollama can still use the GPUs even while Ray workers are running, since they share physical GPU memory. Auto-scaling was therefore skipped — keeping workers always running avoids 30–60s startup latency with no meaningful benefit.

runtime_env over custom images: K3s uses containerd and image builds require elevated access. Ray's runtime_env installs packages on workers on demand and caches them, making it the simpler and recommended approach.

Fixed worker placement per node: The podAntiAffinity rule in the Ray cluster config ensures one worker per physical node, giving each worker exactly 2 GPUs. Without this, both workers could land on the same node, leaving the other node's GPUs unused.

Troubleshooting Reference

Quick Reference Cheatsheet

# Every session start
conda activate ray-client
cd ~/ray-cluster-scripts
./ray-connect.sh

# Check cluster
./ray-status.sh
nvidia-smi

# Run LLM on all 4 GPUs
~/start-vllm-ray.sh
~/start-sglang-ray.sh

# Free GPUs for Ollama
./ray-scale.sh 0
# Restore
./ray-scale.sh 2

# Full cluster restart
./ray-stop.sh && ./ray-start.sh && ./ray-connect.sh

If you followed all the steps properly, you should have your own AI infrastructure ready to use! Run training, fine-tuning and inference jobs from anywhere with ease. You now have sovereignty over your AI; you now own your own AI future. This is just the beginning. The actual journey is in the projects you build with it and how you tweak it to your taste.

KAARL stands for Kosi's Advanced AI Research Laboratory.

Last updated at: June 3, 2026