Skip to content

Slurm Integration Guide

Overview

ORB acts as the power management bridge for elastic/cloud nodes managed by the Slurm Workload Manager by SchedMD. Slurm decides when to scale, based on job queue pressure, and ORB handles how to provision and deprovision cloud resources.

ORB integrates with Slurm's ResumeProgram/SuspendProgram power saving hooks — the same mechanism Slurm uses for elastic cloud bursting.

Prerequisites

  • Slurm cluster (≥ 23.02) with power saving enabled
  • ORB installed on the slurmctld node (or accessible via API)
  • Cloud provider credentials configured in ORB
  • Elastic node definitions in slurm.conf

Cloud Node Model (Dynamic Slots)

ORB treats Slurm cloud nodes as fungible capacity slots:

  • batch-[001-100] defines 100 slots of identical shape
  • Each ResumeProgram call provisions FRESH instances
  • No data residency between suspend/resume cycles
  • Node names are arbitrary handles — batch-005 today may be backed by a completely different EC2 instance than batch-005 yesterday
  • All persistent data should live on shared storage (FSx for Lustre, EFS, NFS)
  • Suspend ALWAYS terminates instances (no stop/start mode)

This matches the AWS ParallelCluster model and SchedMD's official cloud-bursting recommendations.

Configuration Steps

1. Configure ORB for Slurm

Set the scheduler type in your ORB config.json:

{
  "scheduler": {
    "type": "slurm",
    "config_root": "/etc/orb"
  },
  "provider": {
    "providers": [
      {
        "name": "aws_prod",
        "type": "aws",
        "enabled": true,
        "default": true,
        "config": {
          "region": "us-east-1"
        }
      }
    ]
  }
}

2. Configure slurm.conf

Add power saving configuration to slurm.conf:

# Power saving configuration
SuspendProgram=/opt/orb/scripts/suspendProgram.sh
ResumeProgram=/opt/orb/scripts/resumeProgram.sh
SuspendTimeout=120
ResumeTimeout=300
SuspendTime=300
SuspendExcNodes=controller

# Elastic node definitions
NodeName=compute-[001-100] CPUs=4 RealMemory=8192 State=CLOUD
PartitionName=batch Nodes=compute-[001-100] Default=YES MaxTime=INFINITE State=UP

3. Install ORB Scripts

Copy (or symlink) the ORB Slurm scripts:

cp /path/to/orb/infrastructure/scheduler/slurm/scripts/resumeProgram.sh /opt/orb/scripts/
cp /path/to/orb/infrastructure/scheduler/slurm/scripts/suspendProgram.sh /opt/orb/scripts/
chmod +x /opt/orb/scripts/*.sh

4. Hook Configuration File

Slurm spawns ResumeProgram/SuspendProgram as child processes — systemd environment variables on slurmctld do NOT propagate. The hook scripts source a configuration file instead.

Create ${ORB_ROOT_DIR}/slurm_hooks.env (default: /usr/orb/slurm_hooks.env):

# Template to provision when SLURM calls ResumeProgram
SLURM_ORB_TEMPLATE_ID=EC2Fleet-Instant-OnDemand

# Optional overrides
# SLURM_ORB_LOG_DIR=/var/log/orb
# SLURM_ORB_MODE=cli
# SLURM_ORB_API_URL=http://localhost:8000

The scripts source this file before reading any SLURM_ORB_* variables.

5. Environment Variables

Configure these environment variables for ORB's Slurm integration (set them in slurm_hooks.env or the calling environment):

Variable Description Default
ORB_ROOT_DIR ORB installation directory (used for config discovery) /usr/orb
SLURM_ORB_TEMPLATE_ID Template ID for provisioning default (warns)
SLURM_ORB_LOG_DIR ORB log directory /var/log/orb
SLURM_ORB_MODE cli or api cli
SLURM_ORB_API_URL ORB API URL (for api mode) http://localhost:8000
SLURM_ORB_ADDR_POLL_TIMEOUT Seconds to wait for node addresses after the request is submitted; keep below ResumeTimeout. On timeout the script exits 0 and slurmd self-registers 120
SLURM_ORB_ADDR_POLL_INTERVAL Seconds between request status polls 5
SLURM_ORB_RESTD_URL slurmrestd URL (for health checks) Not set
SLURM_ORB_JWT_TOKEN JWT token for slurmrestd auth Not set
SLURM_ORB_RESTD_ALLOW_HTTP Allow plain http:// for SLURM_ORB_RESTD_URL on a non-loopback host (not recommended) 0 (rejected)

6. slurmrestd Integration (Optional)

If slurmrestd is running on your cluster, ORB can use it for health monitoring:

export SLURM_ORB_RESTD_URL=https://slurmctld:6820
export SLURM_ORB_JWT_TOKEN=$(scontrol token lifespan=3600)

Trusted host: SLURM_ORB_RESTD_URL must point to a slurmrestd endpoint you trust — normally the same cluster's slurmctld host, reachable only from ORB's control-plane network. ORB sends the JWT token above to whatever host this URL resolves to, so pointing it at an untrusted host would leak that token.

HTTPS by default, plain HTTP on loopback: slurmrestd has no built-in TLS and normally runs on localhost or a UNIX socket, so ORB allows plain http:// without any extra configuration when the host is loopback (localhost, 127.0.0.0/8, or ::1 — a subdomain like localhost.example.com or an all-interfaces address like 0.0.0.0 does not count). For any other host, plain http:// sends the token and all node/partition data unencrypted, so ORB rejects it unless you explicitly opt in with SLURM_ORB_RESTD_ALLOW_HTTP=1. Only do this on a network you already trust — for anything crossing an untrusted network, put a TLS-terminating proxy in front of slurmrestd and use https:// instead.

This enables: - Cluster health checks via orb system health - Node state monitoring - Partition discovery

Verification

After configuration, verify the integration:

# Check ORB can see the scheduler
orb system status --scheduler slurm

# Test health check (if slurmrestd configured)
orb system health

# Manually trigger a resume (for testing)
/opt/orb/scripts/resumeProgram.sh "compute-001"

# Check node was provisioned
scontrol show node compute-001

CLI Usage with --nodes

The --nodes flag lets ORB work directly with Slurm node names.

Provisioning by Node Name

# Request 3 machines, associating them with SLURM node names
orb machines request EC2Fleet-Instant-OnDemand 3 --nodes "compute-[001-003]"

ORB expands the Slurm hostlist format and stores the node name on each provisioned machine's name field. This enables lookup by node name later.

Terminating by Node Name

# Terminate machines associated with specific SLURM nodes
orb machines terminate --nodes "compute-[001-003]" --force

This resolves node names to machine IDs via storage and terminates them. No in-memory state is needed — node names persist across CLI invocations.

How Node Names Persist

When machines are provisioned with --nodes: 1. The Slurm hostlist is expanded to individual names (e.g. compute-001) 2. Each provisioned machine's name field is set to its assigned node name 3. The name persists in machine storage (survives CLI restarts) 4. terminate --nodes queries storage by name to find machine IDs 5. Provider sync does NOT overwrite explicitly-set node names

AMI/Image Requirements

The provisioned cloud instances must have Slurm pre-installed and configured. ORB provisions the infrastructure; the image handles Slurm membership.

Required on the AMI/image:

  • slurmd installed (same Slurm version as the cluster)
  • slurm.conf configured with correct SlurmctldHost pointing to your controller
  • Munge authentication configured (shared munge key from the cluster)
  • slurmd systemd service enabled to start on boot
  • Network access to slurmctld (typically port 6817)

Reading the Node Name from Instance Tags

ORB tags each provisioned instance with orb:node-name=<SLURM node name>. The compute image's boot script should read this tag to start slurmd with the correct identity:

#!/bin/bash
# /opt/slurm/boot.sh — run on instance boot (e.g. via cloud-init)
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region)

NODE_NAME=$(aws ec2 describe-tags \
  --region "${REGION}" \
  --filters "Name=resource-id,Values=${INSTANCE_ID}" "Name=key,Values=orb:node-name" \
  --query 'Tags[0].Value' --output text)

if [ -n "${NODE_NAME}" ] && [ "${NODE_NAME}" != "None" ]; then
    hostnamectl set-hostname "${NODE_NAME}"
    slurmd -N "${NODE_NAME}"
else
    # Fallback: start slurmd without explicit name (self-registers)
    slurmd
fi

The instance's IAM role needs ec2:DescribeTags permission for this to work.

Recommended:

  • cloud-init or user_data support for dynamic hostname configuration
  • Log forwarding for slurmd logs
  • Health check script for node self-verification

Node Registration Flow

ResumeProgram (N nodes, single batch)
    │
    ▼
ORB batch provisions N fresh EC2 instances
    │
    ▼
resumeProgram.sh polls the request status until addresses are assigned,
then calls: scontrol update NodeAddr=<IP> (for each)
    │
    ▼
Instances boot → slurmd starts → registers with slurmctld
    │
    ▼
SLURM clears POWERING_UP → nodes become IDLE → jobs scheduled

Timing considerations:

Instance Type Typical Boot Time Recommended ResumeTimeout
General (t3, m5) 60-90s 300s
Compute (c5, c6i) 60-90s 300s
GPU (p3, g4) 90-180s 600s
Large/metal 120-300s 900s