Managing InfiniBand Networking

Overview

Crusoe Cloud supports high performance interconnects utilizing NVIDIA Mellanox InfiniBand (IB) networking. The fabric is currently supported for the instance types in the table below:

Instance Type	Number of Infiniband HCAs per Instance	Total InfiniBand Bandwidth (Gbps)
a100-80gb-sxm-ib.8x	8	1600
h100-80gb-sxm-ib.8x	8	3200
h200-141gb-sxm-ib.8x	8	3200

The general workflow, which will be discussed in more detail below, emcompasses selecting an IB network for the list of networks available within a location, and then creating an IB parition from within that network. Finally you will launch instances into that partition. This ensures cluster tenancy within a parition to maximize performance with cluster wide isolation.

InfiniBand VM image

Crusoe Cloud provides a default VM image that comes with all the software libraries and tools necessary to take advantage of InfiniBand through supported ML and HPC frameworks. While you are not required to use this image to use InfiniBand networking, we strongly recommend the ubuntu22.04-nvidia-sxm-docker:latest image for the easiest possible setup. Learn more about images.

info

To support full-performance distributed training on Crusoe’s hypervisor/virtualisation stack, some NCCL configuration changes are required.

NCCL Configuration

Crusoe’s updated hypervisor stack changes the PCI address of devices within the virtual machines. This requires an update to the NCCL XML topology file. The XML file is included by default in Crusoe’s curated image, you can set the following environment variable in /etc/nccl.conf.

NCCL_TOPO_FILE=<PATH_TO_TOPO_FILE>

The *-nvidia-sxm-docker images also comes with a service unit file called crusoe_nccl_topo.service which sets the NCCL_TOPO_FILE environmental variable. If a custom topology file location is used (for example, to customize the topology file), the service should be disabled by running systemctl disable crusoe_nccl_topo.service. The current state of the service may be queried by running systemctl status crusoe_nccl_topo.service.

For NCCL to correctly detect the PCIe topology, the following environment variables must be set in /etc/nccl.conf.

NCCL_IB_MERGE_VFS=0

The NCCL_IB_HCA configuration must be modified to exclude the Ethernet device mlx5_0.

NCCL_IB_HCA=^mlx5_0:1

When using versions of HPC-X older than 2.18, the following argument must be used.

NCCL_IBEXT_DISABLE=1

If you are not using this image, or want to run containers, you must also set all these environment variables:

FROM ...
ENV NCCL_TOPO_FILE=/path/to/nccl_topo.xml
ENV NCCL_IB_MERGE_VFS=0
ENV NCCL_IB_HCA=^mlx5_0:1

NCCL_TOPO_FILE

Provided below are the NCCL Topology files:

Instance Type	NCCL Topology File
a100-80gb-sxm-ib.8x	a100-80gb-sxm-ib.8x
h100-80gb-sxm-ib.8x	h100-80gb-sxm-ib.8x
h200-141gb-sxm-ib.8x	h200-80gb-sxm-ib.8x

InfiniBand Networks

InfiniBand Networks are a logical representation of the physical InfiniBand fabric.

InfiniBand Network limitations

You are limited to a maximum of five InfiniBand partitions on the same InfiniBand network.

Listing InfiniBand Networks and Partitions

CLI

Use the networking ib-networks list and networking ib-partitions list commands to list networks and partitions.

crusoe networking ib-networks list

crusoe networking ib-partitions list

The IDs will be used when attaching a VM to a partition.

UI

In order to view an IB partition via the Crusoe Cloud console:

• Visit the Crusoe Cloud console
• Click the "InfiniBand" tab in the left nav
• View the InfiniBand Networks and Partitions

Terraform

To list existing IB networks using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.

# list ib networks
data "crusoe_ib_networks" "ib_networks" {}

output "crusoe_ib" {
  value = data.crusoe_ib_networks.ib_networks
}

Listing IB partitions is not currently supported in the Crusoe Terraform provider.

Creating InfiniBand Partitions

CLI

Use the networking ib-partitions create command to create a new partition.

crusoe networking ib-partitions create \
  --name my-new-partition \
  --ib-network-id uuid-of-network

UI

In order to create an IB partition via the Crusoe Cloud console:

• Visit the Crusoe Cloud console
• Click the "InfiniBand" tab in the left nav
• Click the "Add Partition" button
• Input a name for the IB partition
• Select the IB network to create the IB parition in
• Click "Create"

Terraform

The following is intended to help get you started using Terraform to provision and InfiniBand Partition and attach an InfiniBand partition to a VM in Crusoe Cloud.

Copy and paste the code below in a text-editor of your choice and name the file main.tf. The example below creates an InfiniBand partition:

// Crusoe Provider
terraform {
  required_providers {
    crusoe = {
      source  = "registry.terraform.io/crusoecloud/crusoe"
    }
  }
}

# list IB networks
data "crusoe_ib_networks" "ib_networks" {}

output "crusoe_ib" {
  value = data.crusoe_ib_networks.ib_networks
}

# create an IB partition to deploy VMs in
resource "crusoe_ib_partition" "my_partition" {
  name = "my-ib-partition"

  # available IB network IDs can be listed by using the output
  # above. alternatively, they can be obtain with the CLI by
  #   crusoe networking ib-networks list
  # copy and paste the network ID that has capacity
  ib_network_id = data.crusoe_ib_networks.ib_networks.ib_networks[0].id
}

name and ib_network_id are required when creating the resource crusoe_ib_partition

name can only include lowercase ascii characters, numbers and -.

ib_network_id is a string which represents the UUID of an existing InfiniBand Network from the output of crusoe networking ib-networks list

Update an existing InfiniBand Partition

CLI

Updating Infiniband partitions is currently unsupported in the Crusoe Cloud CLI.

UI

Updating Infiniband partitions is currently unsupported in the Crusoe Cloud Console.

Terraform

To update an IB network using the Crusoe Terraform provider, you can change the fields of an existing IB network and run terraform apply. The Crusoe Terraform provider will apply the changes to the IB network.

# Update the existing IB partition with the new network ID
resource "crusoe_ib_partition" "my_partition" {
  name = "my-ib-partition" -> "new-ib-partition-name"

  ib_network_id = "ib-network-id" -> "new-ib-network-id"
  project_id = "project-id" -> "new-project-id"
}

Currently, only the "name", "ib_network_id", or "project_id" can be updated. These fields cannot be updated in-place. As a result, modifying any of these fields will replace the resource rather than an in-place update.

Deleting an InfiniBand Partition

info

Warning: deleting an InfiniBand partition is a permanant action that will require re-creation of the partition to recover.

CLI

Infiniband partitions can be deleted in the CLI using the networking ib-partitions delete <id> command.

UI

In order to delete an Infiniband partition via the Crusoe Cloud console:

• Visit the Crusoe Cloud console
• Click the "Infiniband" tab in the left nav
• Navigate to the row of the Infiniband partition you wish to delete
• Click the trash can icon on the far right side of the row
• Click the "Confirm" button

Terraform

An Infiniband partition can be deleted by using the terraform destroy command provided by the Terraform CLI tool.

Launching Instances in an InfiniBand Parition

CLI

Use the compute vms create command to create a new VM, passing in the --ib-partition-id:

crusoe compute vms create \
   --name infiniband-test \
   --location us-east1-a \
   --type a100-80gb-sxm-ib.8x \
   --image ubuntu22.04-nvidia-sxm-docker:latest \
   --ib-partition-id uuid-of-partition \
  ...

UI

In order to connect a VM to an IB partition, you must create the VM and attach it to the appropriate InfiniBand Network and Partition:

• Visit the Crusoe Cloud console
• Click the "Instances" tab in the left nav
• Click "Create instance"
• Select the desired instance type and location, ensuring that the instances and location support InfiniBand
• Select the desired InfiniBand Network
• Select the desired InfiniBand Partition, or create a new InfiniBand Partition
• Continue through the rest of the flow
• Click "Create"

Terraform

Copy and paste the code below in a text-editor of your choice and name the file main.tf. The example below creates an Infiniband partition attached to a VM along with a disk attached:

// Crusoe Provider
terraform {
  required_providers {
    crusoe = {
      source  = "registry.terraform.io/crusoecloud/crusoe"
    }
  }
}

locals {
  my_ssh_key = file("~/.ssh/id_rsa.pub")
}

# attached storage disk
resource "crusoe_storage_disk" "data_disk" {
  name = "data-disk"
  size = "1TiB"
  location = "us-east1-a"
}

# list IB networks
data "crusoe_ib_networks" "ib_networks" {}

output "crusoe_ib" {
  value = data.crusoe_ib_networks.ib_networks
}

# create an IB partition to deploy VMs in
resource "crusoe_ib_partition" "my_partition" {
  name = "my-ib-partition"

  ib_network_id = data.crusoe_ib_networks.ib_networks.ib_networks[0].id
}

# create multiple VMs, all in the same Infiniband partition
resource "crusoe_compute_instance" "my_vm" {
  count = 3

  name = "ib-vm-${count.index}"
  type = "h100-80gb-sxm-ib.8x" // other option: a100-80gb-sxm-ib.8x
  location = "us-east1-a" # IB currently only supported at us-east1-a
  image = "ubuntu22.04-nvidia-sxm-docker:latest" # recommended IB image

  ssh_key = local.my_ssh_key

  host_channel_adapters = [
    {
      ib_partition_id = crusoe_ib_partition.my_partition.id
    }
  ]
  disks = [
    // disk attached at startup
    {
      id = crusoe_storage_disk.data_disk.id
      attachment_type = "data"
      mode = "read-write" // other option: "read-only"
    }
  ]
}

Updating the IB partition on a VM

CLI

Use the compute vms update on a stopped VM to update the --ib-partition-id.

crusoe compute vms update \
   --ib-partition-id uuid-of-new-partition

UI

In order to switch IB networks or partitions in the UI:

• Visit the Crusoe Cloud console
• Click the "Instances" tab in the left nav
• Select the appropriate instance
• Stop the instance if it is not already stopped
• Pick the appropriate IB network and partition from the dropdowns
• Start the instance and verify it is connected to the chosen IB network and partition