Managing your Node Pools

Select your cluster's CMK version. To check a cluster's version, see What's new in CMK v2.

CMK v2
CMK v1

What to know about node pools

A node pool is a group of VMs of the same instance type that join your cluster as worker nodes.
count is the number of nodes you want. The pool keeps that many running. If a VM is deleted, the pool replaces it. If a VM is stopped, the pool adds a new VM in its place and leaves the stopped VM alone.
current is the number of nodes that joined the cluster and became Ready. If nodes couldn't be created, the pool's health issues say why.
Changes to a node pool apply to its running nodes, except changes to the Kubernetes version, ephemeral storage for containerd, and SSH key. Those apply only to nodes created after the change.
Lowering count removes nodes only if you turn on allow scale down. See Scale down a node pool.
The remediation policy controls how Crusoe repairs unhealthy nodes. See AutoClusters in CMK v2.
A node pool can have up to 500 nodes.

You need Crusoe CLI 1.0.10 or later, or Terraform provider 1.5.0 or later.

Create a node pool

CLI
UI
Terraform

Use the crusoe kubernetes nodepools create command:

crusoe kubernetes nodepools create \
  --name my-node-pool \
  --cluster-name my-cluster \
  --type h100-80gb-sxm-ib.8x \
  --count 4 \
  --ib-partition-id <ib-partition-id>

Optional flags:

--node-labels: Labels for every node, such as team=ml,env=prod.
--node-taints: Taints for every node, such as dedicated=gpu:NoSchedule. If your CLI doesn't show this flag, contact support to turn on taints for your organization.
--keyfile: Path to the SSH public key for the nodes. Defaults to the key in your CLI configuration.
--node-pool-version: The worker node version. Defaults to the newest version that's compatible with the cluster.
--ephemeral-storage-for-containerd=true: Use the local NVMe disk for containerd storage.
--consent-mode: The remediation policy, auto, propose, or off. Defaults to auto.
--allow-scale-down=true: Let a lower count remove nodes. Defaults to false.

To see every flag, run crusoe kubernetes nodepools create --help.

Use the crusoe_kubernetes_node_pool resource:

resource "crusoe_kubernetes_node_pool" "my_node_pool" {
  name           = "my-node-pool"
  cluster_id     = crusoe_kubernetes_cluster.my_cluster.id
  type           = "h100-80gb-sxm-ib.8x"
  instance_count = 4
  ssh_key        = file("~/.ssh/id_ed25519.pub")

  # Optional
  # requested_node_labels = { "team" = "ml" }
  # consent_mode          = "auto"
  # update_settings       = { allow_scale_down = true }
}

To pin a worker node version, set version to a version from the Kubernetes Node Pool Versions table of crusoe kubernetes versions list, such as 1.34.5-cmk.3. Don't copy the cluster's version, such as 1.35, into version.

If capacity is short, creation finishes with fewer nodes than count. The pool keeps adding nodes until it reaches count.

View node pools

CLI
UI
Terraform

List the node pools in a cluster:

crusoe kubernetes nodepools list --cluster-name my-cluster

Get one node pool:

crusoe kubernetes nodepools get my-node-pool

The output shows desired count, running count (the API's current), state, and any health issues.

The crusoe_kubernetes_node_pool resource and data source expose the read-only attributes current and health.

Node pool states

State	Meaning
`STATE_PROVISIONING`	The pool is being created.
`STATE_RUNNING`	The pool is running.
`STATE_RUNNING_WITH_ERROR`	The pool is running but is short of `count` because nodes couldn't be created. CMK keeps retrying.
`STATE_UPDATING`	An update is in progress.
`STATE_DELETING`	The pool is being deleted, or a delete failed. Delete it again to retry.

You can update a pool in STATE_RUNNING or STATE_RUNNING_WITH_ERROR. You can delete a pool in those states or in STATE_DELETING.

Health issues

Code	Meaning	What to do
`INSUFFICIENT_CAPACITY`	No capacity for the instance type.	Wait while CMK retries, or lower `count`.
`INSUFFICIENT_QUOTA`	The project's quota doesn't cover the nodes.	Raise the quota or lower `count`.
`INTERNAL_ERROR`	CMK couldn't add nodes. This includes nodes that didn't become Ready within 20 minutes and were deleted.	Wait while CMK retries. If it persists, contact support.
`NODE_NOT_READY`	One or more nodes report NotReady. This doesn't change the pool's state.	Inspect the node with `kubectl describe node`.

Update a node pool

You can change the node count, node labels, node taints, Kubernetes version, ephemeral storage for containerd, remediation policy, and allow scale down.

Changes apply to running nodes right away, except the Kubernetes version and ephemeral storage, which apply only to nodes created after the change. To apply them to existing nodes, see Replace nodes.

When you set node labels or taints, the pool replaces the full set of labels or taints it applied before, on every node. Labels and taints that the pool didn't apply, such as those from the kubelet, add-ons, or kubectl, stay in place. A NoExecute taint immediately evicts pods that don't tolerate it from every node in the pool.

CLI
UI
Terraform

crusoe kubernetes nodepools update my-node-pool \
  --count 6 \
  --node-labels "team=ml,env=prod" \
  --consent-mode auto

Pass only the flags for the settings you want to change. To see every flag, run crusoe kubernetes nodepools update --help.

Edit the crusoe_kubernetes_node_pool resource and run terraform apply:

resource "crusoe_kubernetes_node_pool" "my_node_pool" {
  # ...
  instance_count        = 6
  requested_node_labels = { "team" = "ml", "env" = "prod" }
  consent_mode          = "auto"
}

Changing name, type, ssh_key, cluster_id, subnet_id, project_id, transport_partition_id, nvlink_domain_id, or public_ip_type replaces the whole node pool.

Scale down a node pool

By default, lowering count removes no nodes. The update succeeds and returns a warning that scale-down is disabled.

To let a lower count remove nodes, turn on allow scale down. You can do it in the same update:

crusoe kubernetes nodepools update my-node-pool \
  --allow-scale-down=true \
  --count 2

In Terraform, set update_settings = { allow_scale_down = true }.

The pool removes nodes in this order:

Nodes that have been NotReady for 20 minutes or more
VMs that never joined the cluster
The oldest nodes

For each node, the pool evicts pods and respects PodDisruptionBudgets for up to 5 minutes. It then deletes any remaining pods and the VM.

If the pool already has more nodes than count, turning on allow scale down removes the extra nodes right away.

Don't turn on allow scale down for a pool that the Cluster Autoscaler manages. The autoscaler removes nodes on its own.

Replace nodes

Node pools have no rolling update. To move a node to a new Kubernetes version or ephemeral storage setting, replace it:

Drain the node:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Delete the node's VM, which has the same name as the node:
```
crusoe compute vms delete <node-name>
```

The pool creates a replacement with the current settings. To limit lost capacity, replace one node at a time. Don't stop the VM instead of deleting it. A stopped VM stays in the pool, and the pool adds another VM next to it.

Delete a node pool

Deleting a node pool deletes its VMs without draining them. Move your workloads off the pool first. You must delete every node pool in a cluster before you delete the cluster.

CLI
UI
Terraform

crusoe kubernetes nodepools delete my-node-pool

Remove the crusoe_kubernetes_node_pool resource from your configuration and run terraform apply.

If you have problems creating or deleting node pools, contact support.

What to know about Node Pools

Node pools allow you to group one or more Crusoe Cloud instances of the same type and associate them to a specific cluster control plane, to function as worker nodes.
When creating a node pool, you must specify the number of VMs you want to create via a count field. Once specified, the node pool will maintain this VM count where possible. For example, if you stop or terminate one of the VMs in your node pool, the node pool will provision new VMs up till the count specified.
You may scale up or down your node pool by specifying a new count value. Note that setting a count value lower than the current number does not automatically delete VMs from your node pools. You must manually delete the instances you want removed.
We currently do not allow editing the startup script or image associated with node pools. To install packages or software on node bring-up, we recommend using Daemonsets.

Creating a New Node Pool

CLI
UI
Terraform

Nodepools can be created by using the kubernetes nodepools create command. Nodepools must be created in the context of a specific cluster. Use the '--help' flag for an exhaustive list of options.

crusoe kubernetes nodepools create \
  --name my-first-nodepool \
  --cluster-id 6f8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c \
  --type h100-80gb-sxm-ib.8x \
  --count 4 \
  --ib-partition-id 4c8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c \

You can use the crusoe_kubernetes_node_pool resource to create a new node pools in the context of a specific cluster via Terraform.

terraform {
  required_providers {
    crusoe = {
      source  = "crusoecloud/crusoe"
    }
  }
}

locals {
  my_ssh_key = file("~/.ssh/id_ed25519.pub") # replace with path to your public SSH key if different
}

resource "crusoe_kubernetes_cluster" "my_first_cluster" {
  name = "tf-cluster"
  version = "1.34.5-cmk.x" # Replace with the version you want
  location = "us-east1-a"
  subnet_id = "6f8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c"
  add_ons = ["nvidia_gpu_operator","nvidia_network_operator","crusoe_csi"]
}

resource "crusoe_kubernetes_node_pool" "l40s_nodepool" {
  name = "tf-l40s-nodepool"
  cluster_id = crusoe_kubernetes_cluster.my_first_cluster.id
  instance_count = "4"
  type = "l40s-48gb.10x"
  ssh_key = local.my_ssh_key
  version = crusoe_kubernetes_cluster.my_first_cluster.version
  requested_node_labels = {
    # Optional: Kubernetes Node objects will be labeled with the following key:value pairs
    # "labelkey" = "labelvalue"
  }
}

Viewing Existing Node Pools

Use the kubernetes nodepools list command to list all existing node pools across clusters. You can also use the `kubernetes nodepools get' command to retrieve individual node pool details.

crusoe kubernetes nodepools get <name/id>

Update your Node Pool Template

You can edit the following properties of your node pool:

Change the count of nodes in the node pool
Set the pool to use (or not use) local ephemeral NVMe for containerd storage
Set the labels for your pool. The newly provided labels will overwrite all old labels when new nodes come up.
Set the taints for your pool. The newly provided taints will overwrite all old taints when new nodes come up.
The version of the pool's Kubernetes worker nodes. You can get a list of available versions by running crusoe kubernetes versions list from the Crusoe CLI.

Updating properties of a nodepool will update the nodepool template, but will not perform an in-place upgrade of the existing nodes in the pool. If you scale the nodepool after applying an update, the updated template will only apply to the newly created nodes. To update existing nodes, you must perform a rolling upgrade (see section below).

CLI
UI
Terraform

You may update the number of nodes in your nodepools by using the crusoe kubernetes nodepools update command.

crusoe kubernetes nodepools update <name/id> --count 3

For instructions on how to update additional elements of your nodepool (e.g. ephemeral storage for containerd, nodepool labels), run:

crusoe kubernetes nodepools update -h

resource "crusoe_kubernetes_node_pool" "l40s_nodepool" {
  name           = "tf-l40s-nodepool"
  cluster_id     = crusoe_kubernetes_cluster.my_first_cluster.id

  # Optional: Set the desired instance count
  instance_count = "6" # add 2 more nodes

  # Optional: Set the desired CMK worker node version
  # If not specified, the default is the latest stable version compatible with the cluster
  # List available node pool versions with "crusoe kubernetes versions list"
  version = "1.34.5-cmk.x" # Replace with the version you want

  type    = "l40s-48gb.10x"

  # Optional: Kubernetes Node objects will be labeled with the following key:value pairs
  # requested_node_labels = {
  #   "labelkey" = "labelvalue"
  # }

  # Optional: Use local ephemeral NVMe disks for containerd storage
  # ephemeral_storage_for_containerd = true

}

Updating Nodes in Your Node Pool

To make existing nodes reflect the latest nodepool configuration template, delete the VMs backing the nodes. The node pool will heal by creating new VMs that reflect the new template.

Delete a Node Pool

CLI
UI
Terraform

You can delete nodepools by using the kubernetes nodepools delete command and specifying either the name of ID of the nodepool you want to delete.

crusoe kubernetes nodepools delete <name/id>

Deleting nodepools may be accomplished by removing the desired crusoe_kubernetes_nodepool resource from your terraform configuration.

If you have problems creating or deleting node pools, contact support.

What to know about node pools​

Create a node pool​

View node pools​

Node pool states​

Health issues​

Update a node pool​

Scale down a node pool​

Replace nodes​

Delete a node pool​

What to know about Node Pools​

Creating a New Node Pool​

Viewing Existing Node Pools​

Update your Node Pool Template​

Updating Nodes in Your Node Pool​

Delete a Node Pool​