Deployment Scenarios¶
1. Co-located on slurmctld Node (Simplest)¶
ORB runs directly on the Slurm Workload Manager by SchedMD controller node. The ResumeProgram/SuspendProgram scripts invoke the ORB CLI locally.
┌─────────────────────────────────────┐
│ slurmctld node │
│ │
│ slurmctld ──→ resumeProgram.sh │
│ │ │
│ ▼ │
│ orb CLI (local) │
│ │ │
│ ▼ │
│ AWS / Cloud API │
└─────────────────────────────────────┘
Pros: Simplest setup, no network dependencies. Cons: ORB shares resources with slurmctld.
Setup requirements:
- ORB_ROOT_DIR defaults to /usr/orb — create this directory and place slurm_hooks.env there
- The slurm user (or whichever user slurmctld runs as) must have read access to ${ORB_ROOT_DIR}/slurm_hooks.env and write access to the log directory
- AWS credentials must be accessible to the same user (via instance profile, ~/.aws/, or environment)
# Typical setup
sudo mkdir -p /usr/orb /var/log/orb
sudo chown slurm:slurm /var/log/orb
# Create hook config
cat <<'EOF' | sudo tee /usr/orb/slurm_hooks.env
SLURM_ORB_TEMPLATE_ID=EC2Fleet-Instant-OnDemand
EOF
sudo chmod 644 /usr/orb/slurm_hooks.env
2. Separate Management Node (API Mode)¶
ORB runs on a separate node with its REST API exposed. The Slurm scripts use curl to call the ORB API.
┌──────────────────┐ ┌──────────────────┐
│ slurmctld node │ │ ORB API node │
│ │ │ │
│ resumeProgram.sh │──────▶│ orb serve │
│ (curl to API) │ │ :8000 │
│ │ │ │ │
└──────────────────┘ │ ▼ │
│ AWS / Cloud API │
└──────────────────┘
Configuration:
Pros: Separation of concerns, ORB can serve multiple clusters. Cons: Network dependency, additional infrastructure.
3. Containerized ORB with Slurm Access¶
ORB runs in a container (Docker/Podman) with access to cloud credentials and the Slurm controller.
┌──────────────────┐ ┌─────────────────────┐
│ slurmctld node │ │ Container Host │
│ │ │ ┌───────────────┐ │
│ resumeProgram.sh │──────▶│ │ ORB │ │
│ (curl to API) │ │ │ container │ │
│ │ │ │ :8000 │ │
└──────────────────┘ │ └───────┬───────┘ │
│ ▼ │
│ AWS / Cloud API │
└─────────────────────┘
Pros: Isolated dependencies, easy upgrades, reproducible. Cons: Container networking complexity.
4. Multi-Cluster Setup¶
A single ORB instance manages cloud resources for multiple Slurm clusters, each with its own partition-to-template mapping.
┌─────────────┐
│ Cluster A │──┐
│ slurmctld │ │ ┌──────────────────┐
└─────────────┘ ├────▶│ ORB API │
┌─────────────┐ │ │ (shared) │
│ Cluster B │──┘ │ │ │
│ slurmctld │ │ ▼ │
└─────────────┘ │ AWS / Cloud API │
└──────────────────┘
Configuration: Use different ORB provider instances per cluster, differentiated by provider_name in the template configuration.
Pros: Centralized resource management, single point of cloud credential management. Cons: Single point of failure, more complex configuration.