Skip to main content

Managing Volumes

Managed Slurm clusters include a built-in /home shared volume available to all users. You can attach up to 2 additional Crusoe filesystem disks via the spec.volumes field of your SlurmCluster resource. Each disk is mounted on every login and worker pod at a user-defined path. Volumes can be declared when the cluster is first created or added to an existing cluster at any time.

warning

The existingDiskID field only accepts Crusoe shared-volume (filesystem) disks. Persistent disks are not supported. Verify a disk's type in the Crusoe console or via the Crusoe CLI before using it.

Prerequisites​

  • A Managed Slurm cluster, or an initial SlurmCluster manifest if creating a new one
  • kubectl configured with cluster credentials (see Kubernetes Setup)
  • For existing-disk mode: the UUID of a pre-existing Crusoe shared-volume disk

Volume Modes​

Each entry in spec.volumes must specify exactly one of the following:

Dynamic provisioning (size) — CSO provisions a new Crusoe filesystem disk on your behalf. The disk is created when the volume entry is added and its lifecycle is managed by CSO.

Existing disk (existingDiskID) — You provide the UUID of a pre-existing Crusoe shared-volume disk. CSO mounts the disk but never deletes it, regardless of any reclaimPolicy setting.

Adding Volumes at Cluster Creation​

Include a volumes block in your initial SlurmCluster manifest:

apiVersion: slurm.crusoe.ai/v1alpha1
kind: SlurmCluster
metadata:
  name: my-cluster
  namespace: slurm
spec:
  # ... other fields ...
  volumes:
    # Dynamic: CSO provisions a new 2 TiB disk
    - name: datasets
      size: 2Ti
      mountPath: /mnt/datasets
      reclaimPolicy: Retain # optional; Retain is the default

    # Existing: mount a pre-existing shared-volume disk by UUID
    - name: shared-data
      existingDiskID: 00000000-0000-0000-0000-000000000001
      mountPath: /mnt/shared

Adding Volumes to an Existing Cluster​

Edit your cluster's spec directly with:

kubectl edit slurmcluster <cluster-name> -n slurm

Add entries to spec.volumes using the same format shown above. Save the file to apply the change.

note

Adding a volume entry triggers a rolling restart of all login and worker pods. Pods are restarted one at a time; each new pod will have the disk mounted at the specified path. Running jobs may be interrupted during the restart.

Constraints and Immutability Rules​

FieldMutable?Rules
nameImmutable after creationUsed to name Kubernetes PVC/PV resources. Must be unique within the cluster. Lowercase alphanumeric and hyphens only.
mountPathImmutable after creationCannot be /home (reserved). Cannot be a sub-path or super-path of another volume's mountPath.
existingDiskIDImmutable after creationMust be a valid Crusoe shared-volume UUID.
sizeCan only be increasedMust be a whole number between 1Ti and 10Ti. Decreasing the value is rejected.
reclaimPolicyMutableApplies to dynamic volumes only. Retain (default) keeps the disk after the volume is removed; Delete permanently deletes it.

Additional limits:

  • Maximum 2 volumes per cluster.
  • mountPath values cannot overlap — no entry's path may be a sub-path or super-path of another entry's path.

Checking Volume Status​

After adding a volume, check its provisioning state:

kubectl get slurmcluster <cluster-name> -n slurm -o yaml

Look for status.volumesStatus in the output:

status:
  volumesStatus:
    - name: datasets
      diskID: 00000000-0000-0000-0000-000000000002
      diskName: datasets
      diskSource: crusoe-slurm
      mountPath: /mnt/datasets
      phase: Ready
    - name: shared-data
      diskID: 00000000-0000-0000-0000-000000000001
      diskName: shared-data
      diskSource: customer
      mountPath: /mnt/shared
      phase: Ready

Each entry reports:

FieldDescription
nameMatches the name from spec.volumes
diskIDCrusoe disk UUID (CSO-assigned for crusoe-slurm, user-provided for customer)
diskNameName of the Crusoe disk
diskSourcecrusoe-slurm (CSO-provisioned) or customer (user-provided)
mountPathPath where the disk is mounted in login and worker pods
phaseLifecycle state: Provisioning, Ready, Error, Deleting, Deleted
messageError details when phase is Error

A newly added volume starts in Provisioning and transitions to Ready when the PV and PVC are successfully created. If provisioning fails, the phase shows Error with details in message. When a volume entry is removed, the phase passes through Deleting → Deleted.

Removing a Volume​

Remove the corresponding entry from spec.volumes:

kubectl edit slurmcluster <cluster-name> -n slurm

Delete the entry and save. This triggers a rolling restart of login and worker pods; the disk will no longer be mounted after the restart completes.

What happens to the underlying disk:

  • Dynamic volume, reclaimPolicy: Delete — the Crusoe disk is permanently deleted.
  • Dynamic volume, reclaimPolicy: Retain (default) — the disk is kept in your account and continues to accrue storage charges.
  • Existing disk — the disk is never deleted by CSO, regardless of any settings.
note

If you resize a Crusoe shared-volume disk outside of Slurm (for example, via the Crusoe console or CLI), the change is not automatically reflected in the cluster's PVC. The pods retain access to the full disk capacity, but the PVC may display the previous size.

Next Steps​