Skip to main content

Removing All Support Access

This addendum to Support Access covers how to remove all Crusoe support access to a cluster, going beyond disabling future sessions. Use this procedure when you want to ensure that no Crusoe personnel retain any path into the cluster.

When to Use This Procedure​

The standard Disabling Support Access steps prevent new sessions and—when you uninstall the Helm chart—remove the cluster roles and role bindings that support relies on. However, a support engagement may have left behind workloads that retain access to nodes even after the RBAC roles are gone. Removing all access means revoking RBAC and cleaning up any such workloads.

Step 1: Revoke RBAC Access​

Follow both methods described in Disabling Support Access:

  1. Disable support access via the CLI to prevent new or renewed sessions:

    crusoe kubernetes clusters support-access disable <cluster-name>

  2. Uninstall the Helm chart to immediately remove the support cluster roles and role bindings:

    helm uninstall crusoe-support-roles --namespace crusoe-system
note

Already-minted certificates remain valid until they expire, because Kubernetes does not support certificate revocation. Removing the cluster roles and role bindings (Step 1, item 2) ensures those credentials can no longer authorize any action.

Step 2: Remove Node-Shell and Similar Pods​

During troubleshooting, support engineers may run helper pods—such as node-shell pods—that mount host paths or open a shell onto a node. These pods can provide node-level access independent of the RBAC roles you removed in Step 1, so they must be deleted explicitly.

Crusoe support roles are scoped to the following Crusoe-managed namespaces. Sweep each of them for leftover diagnostic pods:

  • crusoe-system
  • kube-system
  • nvidia-gpu-operator
  • nvidia-network-operator
  • kube-amd-gpu
  • kube-amd-network
  • slinky
  • slurm

  1. Check each Crusoe-managed namespace for any such pods. For example:

    kubectl get pods --namespace crusoe-system

    To check all of the namespaces at once:

    for ns in crusoe-system kube-system nvidia-gpu-operator nvidia-network-operator kube-amd-gpu kube-amd-network slinky slurm; do
      echo "== $ns =="
      kubectl get pods --namespace "$ns"
    done

    Look for pods with names like node-shell-*, debug/shell pods, or any pod you do not recognize as part of normal cluster operation.

  2. Delete any node-shell or similar diagnostic pods you find:

    kubectl delete pod <pod-name> --namespace <namespace>
warning

Only delete pods that you have confirmed are diagnostic or support-related. Deleting pods that are part of normal cluster operation may disrupt your workloads.

Step 3: Verify​

Confirm that support access is fully disabled and no active sessions remain:

crusoe kubernetes clusters support-access get <cluster-name>

Confirm the Helm chart is no longer installed:

helm list --namespace crusoe-system

Confirm no node-shell or diagnostic pods remain in any of the Crusoe-managed namespaces:

for ns in crusoe-system kube-system nvidia-gpu-operator nvidia-network-operator kube-amd-gpu kube-amd-network slinky slurm; do
  echo "== $ns =="
  kubectl get pods --namespace "$ns"
done

Getting Help​

If you have questions about fully revoking support access or identifying which pods are safe to remove, please contact support.