Slurm Operations Guide¶
Template Generation¶
ORB generates templates directly from your slurm.conf partition definitions, ensuring that provisioned instances match the Slurm Workload Manager by SchedMD's expected resource specs.
Basic Usage¶
How It Works¶
- Parses
NodeName=lines for CPUs and RealMemory declarations - Parses
PartitionName=lines to associate partitions with node specs - Maps each partition's CPU/memory requirements to an appropriate AWS instance type
- Generates one template per partition with
template_id= partition name
Example: given this slurm.conf:
NodeName=compute-[001-050] CPUs=4 RealMemory=16000 State=CLOUD
NodeName=gpu-[001-010] CPUs=8 RealMemory=32000 State=CLOUD
PartitionName=batch Nodes=compute-[001-050] Default=YES MaxTime=INFINITE State=UP
PartitionName=gpu Nodes=gpu-[001-010] MaxTime=INFINITE State=UP
ORB generates:
[
{"template_id": "batch", "machine_types": {"t3.xlarge": 1}, "provider_api": "EC2Fleet"},
{"template_id": "gpu", "machine_types": {"t3.2xlarge": 1}, "provider_api": "EC2Fleet"}
]
slurm.conf Path Resolution¶
The path to slurm.conf is resolved in this order:
--slurm-conf /pathCLI flag (highest priority)scheduler.slurm.config_pathin config.jsonSLURM_CONFenvironment variable- Default paths:
/etc/slurm/slurm.conf,/usr/local/etc/slurm.conf,$ORB_ROOT_DIR/slurm.conf
User-Specified Instance Type Preferences¶
For partitions where you want to control instance selection (e.g., for cost optimization or spot diversity), add preferences in config.json:
{
"scheduler": {
"type": "slurm",
"slurm": {
"config_path": "/etc/slurm/slurm.conf",
"partitions": {
"batch": {
"instance_types": ["t3.medium", "t3.large", "m5.large"],
"allocation_strategy": "capacityOptimized"
},
"gpu": {
"instance_types": ["g4dn.xlarge", "g4dn.2xlarge"],
"allocation_strategy": "lowestPrice"
}
}
}
}
}
Rules:
instance_typesis an ordered list (first = highest priority)- All listed types should have CPU >= partition's declared CPUs and Memory >= partition's declared RealMemory
- Partitions without a config entry fall back to auto-mapping from slurm.conf
- Multiple instance types are used as EC2 Fleet overrides for capacity diversification
Generated template output with preferences:
{
"template_id": "batch",
"machine_types": {"t3.medium": 1, "t3.large": 1, "m5.large": 1},
"allocation_strategy": "capacityOptimized",
"provider_api": "EC2Fleet",
"fleet_type": "instant"
}
Validation¶
At generation time, ORB validates that all specified instance types meet the partition's resource requirements. If validation fails, template generation is refused:
ERROR: Template 'batch' validation failed:
- Instance type 't3.micro' (1 vCPU, 1024MB) does not meet partition requirements (CPUs=2, RealMemory=3800)
Fix: Remove undersized instance types from scheduler.slurm.partitions.batch.instance_types
or reduce partition resource requirements in slurm.conf
Use --force to bypass validation (for advanced users).
Overwriting Existing Templates¶
The --force flag both overwrites existing template files and skips instance type validation.
Per-Instance Tagging (orb:node-name)¶
Mechanism¶
When machines are provisioned with --nodes (or via Slurm ResumeProgram):
- Application layer assigns each instance a Slurm node name from the request
- After provisioning, ORB dispatches a
TAG_INSTANCESoperation to the provider - Provider calls
ec2:CreateTagswithorb:node-name=<node_name>on each instance - Compute nodes read this tag on boot to start slurmd with the correct identity
Why It Exists¶
Slurm requires each compute node to register with the name declared in slurm.conf. Since ORB provisions generic instances, the node name assignment happens post-launch via instance tags. The compute AMI's boot script reads the tag and starts slurmd -N <name>.
Provider Requirements¶
- Must support
TAG_INSTANCESoperation type - Must be able to apply per-instance tags after launch
- AWS: requires
ec2:CreateTagspermission on the controller's IAM role - Compute nodes: require
ec2:DescribeTagspermission to read their own tags
Error Handling¶
Tag failures are logged at ERROR level but do NOT fail the overall provisioning request (fire-and-forget). The TAG_INSTANCES operation returns error_result on failure so monitoring can alert on it.
Caveats and Known Limitations¶
Environment¶
| Issue | Detail | Mitigation |
|---|---|---|
| POSIX locale | AL2023 SSM sessions default to LANG=C |
PYTHONUTF8=1 is set automatically in hook scripts and CLI entry point |
| Hook env isolation | systemd env vars don't propagate to ResumeProgram | All config goes in $ORB_ROOT_DIR/slurm_hooks.env |
Networking¶
| Requirement | Reason |
|---|---|
| Security groups: controller ↔ compute all TCP | srun uses ephemeral ports for I/O forwarding |
| Controller on fixed IP/hostname | slurm.conf SlurmctldHost must be reachable from compute |
| DNS resolution | Compute nodes must resolve controller hostname |
Timing¶
| Parameter | Recommended | Reason |
|---|---|---|
ResumeTimeout |
≥ 300s | Instance boot (~60s) + tag propagation (~5s) + tag read with retries (~30s) + slurmd registration |
SuspendTime |
30-300s | How long idle nodes wait before termination; lower = faster cost savings |
SuspendTimeout |
120s | Time for SuspendProgram to complete termination |
| Tag retry window | 30s (15 retries × 2s) | Built into boot script; covers EC2 tag propagation delay |
IAM Requirements¶
Controller node (slurmctld) IAM role:
ec2:RunInstances,ec2:CreateFleet,ec2:TerminateInstancesec2:CreateLaunchTemplate,ec2:CreateLaunchTemplateVersionec2:CreateTags(for per-instance tagging)ec2:DescribeInstances,ec2:DescribeFleetsiam:PassRole(to attach instance profile to compute nodes)ssm:GetParameter(if using SSM for AMI resolution)
Compute node IAM role (instance profile):
ec2:DescribeTags(read ownorb:node-nametag)ssm:GetParameter(if fetching munge key / slurm.conf from SSM)
The instance profile must be specified in config:
"provider_defaults": {
"aws": {
"template_defaults": {
"iam_instance_profile": "orb-compute-instance-profile"
}
}
}
Slurm Resource Matching¶
- Slurm requires all nodes in a partition to match the declared
CPUsandRealMemory - Instances with MORE resources than declared are accepted (Slurm uses declared values for scheduling)
- Instances with FEWER resources will fail
slurmdregistration - Generated templates should use a single instance type per partition, OR multiple types that ALL exceed the partition's declared resources
orb templates generatevalidates this automatically
Tag Race Condition¶
EC2 tags are eventually consistent. Between CreateTags and the instance boot script reading the tag, there is a propagation window (typically < 5s, worst case ~30s).
The recommended boot script pattern:
# Retry loop for tag propagation
for i in $(seq 1 15); do
NODE_NAME=$(aws ec2 describe-tags --region "${REGION}" \
--filters "Name=resource-id,Values=${INSTANCE_ID}" "Name=key,Values=orb:node-name" \
--query 'Tags[0].Value' --output text 2>/dev/null)
if [ -n "${NODE_NAME}" ] && [ "${NODE_NAME}" != "None" ]; then
break
fi
sleep 2
done
Compute AMI Requirements¶
Base Packages¶
- Amazon Linux 2023 (al2023) or Ubuntu 22.04+
- Slurm 24.05.5+ compiled with
--with-systemdfor cgroup/v2 support - Build dependencies:
systemd-devel,dbus-devel,munge-devel - Runtime:
munge,slurm-slurmd
Configuration Files¶
| File | Purpose |
|---|---|
/etc/slurm/slurm.conf |
Must match controller's config (SlurmctldHost, partitions) |
/etc/slurm/cgroup.conf |
cgroup/v2 configuration for resource enforcement |
/etc/munge/munge.key |
Shared authentication key (must match controller) |
Boot Script¶
The AMI must include a boot script (via cloud-init or systemd oneshot) that:
- Reads
orb:node-nameEC2 tag (with retry loop for propagation) - Sets hostname to the Slurm node name
- Optionally fetches
slurm.confandmunge.keyfrom SSM Parameter Store - Starts munge
- Starts slurmd with
-N <node_name>
Example minimal boot script:
#!/bin/bash
set -euo pipefail
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region)
# Wait for orb:node-name tag
for i in $(seq 1 15); do
NODE_NAME=$(aws ec2 describe-tags --region "${REGION}" \
--filters "Name=resource-id,Values=${INSTANCE_ID}" "Name=key,Values=orb:node-name" \
--query 'Tags[0].Value' --output text 2>/dev/null)
[ -n "${NODE_NAME}" ] && [ "${NODE_NAME}" != "None" ] && break
sleep 2
done
if [ -z "${NODE_NAME}" ] || [ "${NODE_NAME}" = "None" ]; then
echo "ERROR: Could not read orb:node-name tag after 30s" >&2
exit 1
fi
# Set hostname and start SLURM
hostnamectl set-hostname "${NODE_NAME}"
systemctl start munge
slurmd -N "${NODE_NAME}"
cgroup.conf Example¶
Verification Checklist¶
- [ ]
slurmd -Creports correct CPUs/RealMemory matching slurm.conf - [ ] munge auth succeeds:
munge -n | ssh controller unmunge - [ ] slurmd can reach slurmctld on port 6817
- [ ] Instance profile has
ec2:DescribeTagspermission - [ ] Boot script handles tag propagation delay (retry loop)
- [ ] Security groups allow controller ↔ compute traffic (all TCP)