Slurm Integration Guide¶
Overview¶
ORB acts as the power management bridge for elastic/cloud nodes managed by the Slurm Workload Manager by SchedMD. Slurm decides when to scale, based on job queue pressure, and ORB handles how to provision and deprovision cloud resources.
ORB integrates with Slurm's ResumeProgram/SuspendProgram power saving hooks — the same mechanism Slurm uses for elastic cloud bursting.
Prerequisites¶
- Slurm cluster (≥ 23.02) with power saving enabled
- ORB installed on the slurmctld node (or accessible via API)
- Cloud provider credentials configured in ORB
- Elastic node definitions in
slurm.conf
Cloud Node Model (Dynamic Slots)¶
ORB treats Slurm cloud nodes as fungible capacity slots:
batch-[001-100]defines 100 slots of identical shape- Each ResumeProgram call provisions FRESH instances
- No data residency between suspend/resume cycles
- Node names are arbitrary handles —
batch-005today may be backed by a completely different EC2 instance thanbatch-005yesterday - All persistent data should live on shared storage (FSx for Lustre, EFS, NFS)
- Suspend ALWAYS terminates instances (no stop/start mode)
This matches the AWS ParallelCluster model and SchedMD's official cloud-bursting recommendations.
Configuration Steps¶
1. Configure ORB for Slurm¶
Set the scheduler type in your ORB config.json:
{
"scheduler": {
"type": "slurm",
"config_root": "/etc/orb"
},
"provider": {
"providers": [
{
"name": "aws_prod",
"type": "aws",
"enabled": true,
"default": true,
"config": {
"region": "us-east-1"
}
}
]
}
}
2. Configure slurm.conf¶
Add power saving configuration to slurm.conf:
# Power saving configuration
SuspendProgram=/opt/orb/scripts/suspendProgram.sh
ResumeProgram=/opt/orb/scripts/resumeProgram.sh
SuspendTimeout=120
ResumeTimeout=300
SuspendTime=300
SuspendExcNodes=controller
# Elastic node definitions
NodeName=compute-[001-100] CPUs=4 RealMemory=8192 State=CLOUD
PartitionName=batch Nodes=compute-[001-100] Default=YES MaxTime=INFINITE State=UP
3. Install ORB Scripts¶
Copy (or symlink) the ORB Slurm scripts:
cp /path/to/orb/infrastructure/scheduler/slurm/scripts/resumeProgram.sh /opt/orb/scripts/
cp /path/to/orb/infrastructure/scheduler/slurm/scripts/suspendProgram.sh /opt/orb/scripts/
chmod +x /opt/orb/scripts/*.sh
4. Hook Configuration File¶
Slurm spawns ResumeProgram/SuspendProgram as child processes — systemd environment variables on slurmctld do NOT propagate. The hook scripts source a configuration file instead.
Create ${ORB_ROOT_DIR}/slurm_hooks.env (default: /usr/orb/slurm_hooks.env):
# Template to provision when SLURM calls ResumeProgram
SLURM_ORB_TEMPLATE_ID=EC2Fleet-Instant-OnDemand
# Optional overrides
# SLURM_ORB_LOG_DIR=/var/log/orb
# SLURM_ORB_MODE=cli
# SLURM_ORB_API_URL=http://localhost:8000
The scripts source this file before reading any SLURM_ORB_* variables.
5. Environment Variables¶
Configure these environment variables for ORB's Slurm integration
(set them in slurm_hooks.env or the calling environment):
| Variable | Description | Default |
|---|---|---|
ORB_ROOT_DIR |
ORB installation directory (used for config discovery) | /usr/orb |
SLURM_ORB_TEMPLATE_ID |
Template ID for provisioning | default (warns) |
SLURM_ORB_LOG_DIR |
ORB log directory | /var/log/orb |
SLURM_ORB_MODE |
cli or api |
cli |
SLURM_ORB_API_URL |
ORB API URL (for api mode) | http://localhost:8000 |
SLURM_ORB_ADDR_POLL_TIMEOUT |
Seconds to wait for node addresses after the request is submitted; keep below ResumeTimeout. On timeout the script exits 0 and slurmd self-registers |
120 |
SLURM_ORB_ADDR_POLL_INTERVAL |
Seconds between request status polls | 5 |
SLURM_ORB_RESTD_URL |
slurmrestd URL (for health checks) | Not set |
SLURM_ORB_JWT_TOKEN |
JWT token for slurmrestd auth | Not set |
SLURM_ORB_RESTD_ALLOW_HTTP |
Allow plain http:// for SLURM_ORB_RESTD_URL on a non-loopback host (not recommended) |
0 (rejected) |
6. slurmrestd Integration (Optional)¶
If slurmrestd is running on your cluster, ORB can use it for health monitoring:
export SLURM_ORB_RESTD_URL=https://slurmctld:6820
export SLURM_ORB_JWT_TOKEN=$(scontrol token lifespan=3600)
Trusted host: SLURM_ORB_RESTD_URL must point to a slurmrestd endpoint
you trust — normally the same cluster's slurmctld host, reachable only from
ORB's control-plane network. ORB sends the JWT token above to whatever host
this URL resolves to, so pointing it at an untrusted host would leak that
token.
HTTPS by default, plain HTTP on loopback: slurmrestd has no built-in TLS
and normally runs on localhost or a UNIX socket, so ORB allows plain
http:// without any extra configuration when the host is loopback
(localhost, 127.0.0.0/8, or ::1 — a subdomain like
localhost.example.com or an all-interfaces address like 0.0.0.0 does
not count). For any other host, plain http:// sends the token and all
node/partition data unencrypted, so ORB rejects it unless you explicitly
opt in with SLURM_ORB_RESTD_ALLOW_HTTP=1. Only do this on a network you
already trust — for anything crossing an untrusted network, put a
TLS-terminating proxy in front of slurmrestd and use https:// instead.
This enables:
- Cluster health checks via orb system health
- Node state monitoring
- Partition discovery
Verification¶
After configuration, verify the integration:
# Check ORB can see the scheduler
orb system status --scheduler slurm
# Test health check (if slurmrestd configured)
orb system health
# Manually trigger a resume (for testing)
/opt/orb/scripts/resumeProgram.sh "compute-001"
# Check node was provisioned
scontrol show node compute-001
CLI Usage with --nodes¶
The --nodes flag lets ORB work directly with Slurm node names.
Provisioning by Node Name¶
# Request 3 machines, associating them with SLURM node names
orb machines request EC2Fleet-Instant-OnDemand 3 --nodes "compute-[001-003]"
ORB expands the Slurm hostlist format and stores the node name on each
provisioned machine's name field. This enables lookup by node name later.
Terminating by Node Name¶
# Terminate machines associated with specific SLURM nodes
orb machines terminate --nodes "compute-[001-003]" --force
This resolves node names to machine IDs via storage and terminates them. No in-memory state is needed — node names persist across CLI invocations.
How Node Names Persist¶
When machines are provisioned with --nodes:
1. The Slurm hostlist is expanded to individual names (e.g. compute-001)
2. Each provisioned machine's name field is set to its assigned node name
3. The name persists in machine storage (survives CLI restarts)
4. terminate --nodes queries storage by name to find machine IDs
5. Provider sync does NOT overwrite explicitly-set node names
AMI/Image Requirements¶
The provisioned cloud instances must have Slurm pre-installed and configured. ORB provisions the infrastructure; the image handles Slurm membership.
Required on the AMI/image:
- slurmd installed (same Slurm version as the cluster)
- slurm.conf configured with correct
SlurmctldHostpointing to your controller - Munge authentication configured (shared munge key from the cluster)
- slurmd systemd service enabled to start on boot
- Network access to slurmctld (typically port 6817)
Reading the Node Name from Instance Tags¶
ORB tags each provisioned instance with orb:node-name=<SLURM node name>.
The compute image's boot script should read this tag to start slurmd with
the correct identity:
#!/bin/bash
# /opt/slurm/boot.sh — run on instance boot (e.g. via cloud-init)
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region)
NODE_NAME=$(aws ec2 describe-tags \
--region "${REGION}" \
--filters "Name=resource-id,Values=${INSTANCE_ID}" "Name=key,Values=orb:node-name" \
--query 'Tags[0].Value' --output text)
if [ -n "${NODE_NAME}" ] && [ "${NODE_NAME}" != "None" ]; then
hostnamectl set-hostname "${NODE_NAME}"
slurmd -N "${NODE_NAME}"
else
# Fallback: start slurmd without explicit name (self-registers)
slurmd
fi
The instance's IAM role needs ec2:DescribeTags permission for this to work.
Recommended:
- cloud-init or user_data support for dynamic hostname configuration
- Log forwarding for slurmd logs
- Health check script for node self-verification
Node Registration Flow¶
ResumeProgram (N nodes, single batch)
│
▼
ORB batch provisions N fresh EC2 instances
│
▼
resumeProgram.sh polls the request status until addresses are assigned,
then calls: scontrol update NodeAddr=<IP> (for each)
│
▼
Instances boot → slurmd starts → registers with slurmctld
│
▼
SLURM clears POWERING_UP → nodes become IDLE → jobs scheduled
Timing considerations:
| Instance Type | Typical Boot Time | Recommended ResumeTimeout |
|---|---|---|
| General (t3, m5) | 60-90s | 300s |
| Compute (c5, c6i) | 60-90s | 300s |
| GPU (p3, g4) | 90-180s | 600s |
| Large/metal | 120-300s | 900s |