# Crusoe Cloud
> The Planet's Favorite Cloud
---
# Overview
Crusoe provides flexible options for running AI workloads. You can train, deploy, and run inference workloads using Crusoe's managed infrastructure or your own infrastructure.
**Ready to build?**
Use one of our [Quickstarts →](/quickstart/overview)
---
### Managed AI
Use Crusoe's [Managed AI services](/managed-ai/overview) to enable Crusoe to provision and manage your workloads with native expertise.
- [Serverless Inference](/serverless-inference/overview): Run inference workloads with Crusoe's Managed AI services.
- [Self-Serve Deployments](/self-serve-deployments/overview): Spin up dedicated, self-serve inference deployments on managed infrastructure.
- [Serverless Fine-Tuning](/serverless-fine-tuning/overview): Fine-tune an open model using a LoRA-based supervised training workflow through the Intelligence Foundry.
### Infrastructure Cloud
Use our [Infrastructure Cloud](/infrastructure-cloud/overview) to spin up a GPU cluster for training, serving, or data processing workloads, and manage your own infrastructure on Crusoe.
- [Spin up your first GPU cluster](/quickstart/spin-up-gpu-cluster): Provision a multi-node GPU cluster with Kubernetes (CMK) or Slurm for training, inference, and data processing.
- [Get started with Terraform](/infrastructure-cloud/terraform): Manage Crusoe infrastructure programmatically using Terraform.
- [Install the CLI](/installing-the-cli): Install and authenticate the Crusoe CLI.
---
# Create an account
Create an account and add billing information to start using Crusoe Cloud. Use Google or Github for the quickest signup.
## 1. Sign up for Crusoe Cloud
Sign up for Crusoe Cloud based on whether you plan to manage your own infrastructure or run inference on Crusoe's managed infrastructure:
- **Create a Managed AI account →**
Run inference on Crusoe's managed infrastructure—with no GPU infrastructure to provision or operate yourself.
- **Create an Infrastructure Cloud account →**
Get direct access to Crusoe GPU compute, storage, and networking, managed by you through VMs, Kubernetes (CMK), or Slurm.
:::note
You can switch between accounts in the console regardless of which account type you initially used for sign up.
:::
## 2. Verify your email address
If you signed up with Google or Github, your email address is already verified—skip to [Enable billing](#3-enable-billing).
If you signed up with your email address, Crusoe Cloud sends a six-digit verification code to that address:
1. Check your inbox for an email from Crusoe Cloud containing the code.
2. Enter the code on the verification screen in the [console](https://console.crusoecloud.com).
If you don't receive a code, check your spam folder before requesting a new one. If you still can't verify your email address, [contact support](/resources/support).
### Considerations
- You can't sign in until your email address is verified.
- Each code expires five minutes after it's sent. If yours expires, request a new one from the verification screen.
- After five incorrect entries, the code stops working. Request a new code and try again.
## 3. Enable billing
To provision resources or use the [managed inference service](/serverless-inference/overview), you must enable billing on your account. You will need a valid, non-prepaid credit card to proceed.
**UI:**
To enable billing via the [console](https://console.crusoecloud.com):
1. From the [console](https://console.crusoecloud.com), click **Admin** in the bottom-left corner.
2. Click **Billing** > **[Payments](https://console.crusoecloud.com/billing/payments)** in the left navigation.
3. Click **Add payment method**.
4. Enter your credit card information.
5. Click **Save**.
If you can't enable billing, please [contact support](/resources/support) with additional details.
## Projects and organizations
When you sign up for Crusoe Cloud, a default organization and project are created for you. Projects and organizations are used to organize your resources—such as models, datasets, and compute instances—and manage access to them.
---
# Install the Crusoe Cloud MCP Server
The Crusoe Cloud Model Context Protocol (MCP) Server enables AI models and agents to query and analyze your Crusoe AI infrastructure.
:::info
This feature is currently in Preview.
:::
## Prerequisites
- **Node.js**: v18 or higher.
- **Crusoe Credentials**: An API key pair configured in `~/.crusoe/config` (standard Crusoe CLI [configuration](/installing-the-cli)).
- **MCP Client**: An MCP-compliant client, such as [Claude](https://claude.ai/download).
## Installation
Choose the instructions for your MCP client below.
### Claude Code
If you have Claude Code installed, add the server directly:
```sh
claude mcp add crusoe-cloud -- npx -y @crusoeai/cloud-mcp
```
### Claude Desktop
The simplest way to install in Claude Desktop is to download the bundled extension (`.mcpb`) and open it. Claude Desktop will prompt you to install it as an extension.
Download Crusoe Cloud MCP (.mcpb)
Once downloaded, double-click the file (or open it from Claude Desktop's **Settings → Extensions**) to install.
Manual configuration (alternative)
You can also add the server definition directly to your Claude Desktop configuration file:
- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`
```json
{
"mcpServers": {
"crusoe-cloud": {
"command": "npx",
"args": ["-y", "@crusoeai/cloud-mcp"]
}
}
}
```
Restart Claude Desktop to apply changes.
### Other MCP Clients
For other MCP-compliant clients (e.g., Cursor, Zed, or custom tools), use the following standard connection details:
- **Type**: `stdio`
- **Command**: `npx`
- **Args**: `-y`, `@crusoeai/cloud-mcp`
## Authentication
The MCP server inherits authentication from your local Crusoe configuration file (`~/.crusoe/config`). It does not store credentials separately.
### Default Setup
If you already use the Crusoe CLI or Terraform provider and have a `[default]` profile in `~/.crusoe/config`, no action is required.
### Configuration via Environment Variables
To use a specific profile from `~/.crusoe/config`, or a specific project, add environment variables when you start a given server instance:
```sh
claude mcp add crusoe-cloud \
-e CRUSOE_PROFILE=staging \
-e CRUSOE_PROJECT_ID=48ca861f-d5ef-4faf-b141-0e4f786c587f \
-- npx -y @crusoeai/cloud-mcp
```
Alternatively, pass them as environment variables in your configuration:
```json
{
"mcpServers": {
"crusoe-cloud": {
"command": "npx",
"args": ["-y", "@crusoeai/cloud-mcp"],
"env": {
"CRUSOE_PROFILE": "staging"
}
}
}
}
```
## Usage Guidelines
- **Security Note: Interaction with CLI**: While this MCP server is strictly read-only, AI agents with shell or terminal capabilities may still modify your Crusoe infrastructure by invoking your installed `crusoe` CLI directly. The read-only constraint applies only to tools provided by this server, not to the agent's broader environment access.
- **Context Efficiency**: Large lists are truncated (default 50 items) to prevent context window overflow. Use filters or specific `get_` tools for granular data.
- **Rate Limiting**: A client-side rate limit of **60 requests per minute** is enforced to prevent accidental API quota exhaustion during AI retry loops. If this limit is hit, the assistant will be instructed to wait before retrying.
## Available Tools
The server exposes read-only tools organized by resource category. All write operations (create, update, delete) are disabled by design.
### Core & Discovery
- `get_user_identity`: Get details of the currently authenticated user.
- `get_organization`: Get organization details.
- `list_capacities`: List available instance types (e.g., `a100.8x`) and availability.
- `get_current_project` / `list_available_projects`: Inspect and list accessible projects.
- `use_project`: Switch the active project for the current session.
- `get_resource_relationships`: Generates a topology graph of resources (VMs, disks, VPCs) and how they connect. _Useful for initial exploration._
### Compute
- `list_vms` / `get_vm`: View virtual machines and their details.
- `list_instance_groups` / `get_instance_group`: View managed instance groups.
- `list_instance_templates` / `get_instance_template`: Inspect VM templates.
- `list_group_instances`: List VMs belonging to a specific group.
### Storage & Images
- `list_disks` / `get_disk`: View block storage volumes.
- `list_disk_snapshots` / `get_disk_snapshot`: View storage snapshots.
- `list_images` / `get_image`: View public VM images.
- `list_custom_images` / `get_custom_image`: View user-created custom images.
### Networking
- `list_vpc_networks` / `get_vpc_network`: View VPCs.
- `list_vpc_subnets` / `get_vpc_subnet`: View subnets.
- `list_firewall_rules` / `get_firewall_rule`: Inspect security rules.
- `list_load_balancers` / `get_load_balancer`: View traffic distribution configurations.
### Orchestration (Kubernetes)
- `list_kubernetes_clusters` / `get_kubernetes_cluster`: View CMK clusters.
- `list_kubernetes_node_pools` / `get_kubernetes_node_pool`: Inspect node pools.
- `list_kubernetes_node_pool_instances`: List specific nodes within a pool.
- `list_kubernetes_versions`: Check supported K8s versions.
### Operations & Billing
- `list_audit_logs`: Detailed logs of who did what and when.
- `list_org_quotas` / `list_project_quotas`: Check resource limits vs. usage.
- `get_gpu_tracking`: Monitor GPU utilization and reservation capacity.
- `get_usage_by_project`: Retrieve usage data for billing analysis.
## Troubleshooting
**"Crusoe credentials not found"**: Ensure `~/.crusoe/config` exists and contains a valid `[default]` profile. You can verify this by running `crusoe whoami` in your terminal.
**Rate Limit Errors**: If you see "Rate limit exceeded," the assistant is making too many rapid queries. Ask the assistant to batch its requests or pause briefly.
**Connection Errors**: The server communicates with `api.crusoecloud.com`. Ensure your machine has outbound HTTPS access to this endpoint.
## Disclaimer
**This is an experimental tool. Use with caution and at your own risk.**
This MCP server enables AI assistants to query your Crusoe Cloud infrastructure through natural language. While the server performs **read-only operations**, you should be aware of the following:
### Data Privacy
- **API responses containing your infrastructure data will be sent to your chosen AI assistant** (e.g., Claude, ChatGPT, or other services)
- Your Crusoe API credentials remain local on your machine and are never sent to Crusoe or third-party AI services
- Be mindful that VM names, network configurations, and other infrastructure details may contain sensitive business information
### AI Decision-Making
- **Always verify information before taking action.** AI assistants may misinterpret infrastructure data or provide incorrect recommendations
- This tool provides information only — it cannot and does not take any destructive actions
- Never rely solely on AI-generated advice for critical infrastructure decisions
### No Warranty
This software is provided "AS IS" without warranty of any kind. Crusoe makes no guarantees regarding:
- Stability or reliability of the tool
- Accuracy of data filtering or response formatting
- Compatibility with all AI assistants or future API changes
---
# Quickstarts
Follow the quickstarts in this guide to train models, run inference, or stand up your own Crusoe infrastructure. First, select a tab below based on how you'd like to host your work:
- **[Managed AI](/managed-ai/overview)**: Run inference or train a model and have Crusoe manage the infrastructure for you.
- **[Infrastructure Cloud](/infrastructure-cloud/overview)**: Provision and operate your own compute on Crusoe.
**Managed AI:**
| Quickstart | Description |
| ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- |
| [Serverless Inference](/serverless-inference) | Generate an API key and send requests to hosted models over an OpenAI-compatible API |
| [Self-Serve Deployments](/self-serve-deployments/overview) | Run inference on reserved GPU capacity that Crusoe manages end-to-end |
| [Serverless Fine-Tuning](/managed-ai/serverless-fine-tuning) | Upload a dataset, launch a LoRA fine-tuning job, and download the resulting checkpoint—no GPU infrastructure required |
**Infrastructure Cloud:**
| Quickstart | Description |
| ------------------------------------------------------------- | -------------------------------------------------------------- |
| [Install the CLI](/installing-the-cli) | Install and authenticate the Crusoe CLI |
| [Get started with Terraform](/infrastructure-cloud/terraform) | Manage infrastructure programmatically using Terraform |
| [Create a VM](/quickstart/creating-a-vm) | Provision a GPU or CPU VM and connect via SSH to run workloads |
| [Spin up a GPU cluster](/quickstart/spin-up-gpu-cluster) | Provision a multi-node GPU cluster using Kubernetes or Slurm |
---
# Serverless Inference
Send requests to Crusoe-hosted models over an OpenAI-compatible API or interact with them directly using the Intelligence Foundry's [chat interface](https://console.crusoecloud.com/foundry/chat/new). Crusoe handles serving, scaling, and optimization—so you don't have to manage any GPU infrastructure.
## 1. Log in or create an account
Log in to the [Crusoe Cloud Console](https://console.crusoecloud.com) or [Create an account](/create-an-account).
After you log in, switch to the **Intelligence Foundry** app in the bottom-left of the [console](https://console.crusoecloud.com/).
## 2. Generate an API key
To create an API key via the [console](https://console.crusoecloud.com):
1. From the [console](https://console.crusoecloud.com), click **Admin** in the bottom-left corner.
2. Select **Security** > **[Intelligence API keys](https://console.crusoecloud.com/security/inference-api-keys)** from the left navigation.
3. Click **Create**.
4. (Optional) Enter an alias for your key.
5. (Optional) Enter an expiration date for your key.
6. Copy the **API key**. Make sure that you save the key in a secure location before leaving the page.
For more information, see [Manage API keys](/identity-and-security/managing-api-keys).
## 3. (Optional) Browse available models
You can use the OpenAI-API compatible endpoint at `api.inference.crusoecloud.com` to access the models below for [Serverless Inference](/quickstart/getting-started-with-serverless-inference). You can also interact with all of the models using the Intelligence Foundry's [chat interface](https://console.crusoecloud.com/foundry/chat/new). All Meta models provided by Crusoe are "Built with Llama".
For each model's pricing information, see [pricing](https://www.crusoe.ai/cloud/pricing#Serverless-Inference).
| MODEL | PROVIDER | TYPE | CONTEXT LENGTH | LICENSE | ACCEPTABLE USE POLICY |
| ------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| [deepseek-ai/DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324) | DeepSeek | instruct | 160k | [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/main/LICENSE) | |
| [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) | DeepSeek | instruct | 1M | [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/LICENSE) | |
| [deepseek-ai/DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) | DeepSeek | instruct | 1M | [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813/blob/main/LICENSE) | |
| [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31B-it) | Google | instruct | 262k | [Apache License 2.0](https://ai.google.dev/gemma/apache_2) | |
| [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) | Meta | instruct | 128k | [Llama 3.3 Community License Agreement](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/LICENSE) | [Llama 3.3 Acceptable Use Policy](https://www.llama.com/llama3_3/use-policy/) |
| [moonshotai/Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) | Moonshot | instruct | 256K | [Modified MIT License](https://huggingface.co/moonshotai/Kimi-K2.6/blob/main/LICENSE) | |
| [nvidia/Nemotron-3-Nano-30B-A3B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8) | NVIDIA | instruct | 262k | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [nvidia/Nemotron-3-Nano-Omni-Reasoning-30B-A3B](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8) | NVIDIA | instruct | 262k | [NVIDIA Open Model Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/) | |
| [nvidia/Nemotron-3-Super-120B-A12B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) | NVIDIA | instruct | 262k | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [nvidia/Nemotron-3-Ultra-550B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4) | NVIDIA | instruct | 262k | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [nvidia/Nemotron-3-VoiceChat](https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B) | NVIDIA | speech-to-speech | 131k | [NVIDIA Software and Model Evaluation License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-and-model-evaluation-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-and-model-evaluation-license/) |
| [nvidia/nemotron-3.5-lightning-30b-a3b](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) | NVIDIA | instruct | 1M | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) | OpenAI | instruct | 128k | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) | [Acceptable Use Policy](https://huggingface.co/openai/gpt-oss-120b/blob/main/USAGE_POLICY) |
| [qwen/Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) | Qwen | instruct | 131k | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE) |
| [qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | Qwen | instruct | 256k | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE) |
| [zai/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1) | Z.ai | instruct | 202k | [MIT License](https://huggingface.co/zai-org/GLM-5.1/blob/main/LICENSE) | |
| [zai/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) | Z.ai | instruct | 256k | [MIT License](https://huggingface.co/zai-org/GLM-5.2/blob/main/LICENSE) | |
| [zai/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) | Z.ai | instruct | 1M | [MIT License](https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE) | |
| [zai/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) | Z.ai | instruct | 1M | [MIT License](https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/LICENSE) | |
## 4. Send a request
Use the OpenAI-compatible endpoint at `api.inference.crusoecloud.com`. The example below queries `meta-llama/Llama-3.3-70B-Instruct`:
Need higher rate limits or reserved capacity?
When interacting with a Serverless endpoint, you might receive a `429 Too Many Requests` response due to rate limits. If you need to exceed the default rate limits, you can:
- [Contact us](https://www.crusoe.ai/contact-sales) for a rate limit increase. This is recommended if you expect your initial launch traffic to exceed the default limits.
- If you need reserved inference capacity, use [Self-Serve Deployments](/self-serve-deployments/overview).
```python
import os
from openai import OpenAI
client = OpenAI(
api_key=os.getenv("CRUSOE_API_KEY"),
base_url="https://api.inference.crusoecloud.com/v1",
)
completion = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful, concise assistant."},
{"role": "user", "content": "Who is Robinson Crusoe?"},
],
)
print(completion.choices[0].message.content)
```
**Additional resources:**
- [Serverless Inference overview](/serverless-inference/overview)
- [Available models and pricing](/serverless-inference/usage-billing-models)
- [Inference metrics](/serverless-inference/inference-metrics)
---
# Deploy with self-serve deployments
Self-serve deployments give you reserved inference capacity on Crusoe's optimized inference engine and managed infrastructure. You choose a base model (or a fine-tuned adapter) and a deployment configuration, and Crusoe handles engine selection, tuning, autoscaling, and rate limiting. You get predictable performance, dedicated throughput, no shared rate limits, and per-GPU-hour billing you control.
## Prerequisites
- Install the OpenAI Python client (`pip install openai httpx`), if you use the Python examples.
- For Low-Rank Adaptation (LoRA) adapters, bring a checkpoint from a successful training job completed with [Serverless Fine-Tuning](/serverless-fine-tuning/overview).
## 1. Log in or create an account
Log in to the [Crusoe Cloud Console](https://console.crusoecloud.com) or [Create an account](/create-an-account).
After you log in, switch to the **Intelligence Foundry** app in the bottom-left of the [console](https://console.crusoecloud.com/).
## 2. Generate an API key and authenticate
To create an API key through the [console](https://console.crusoecloud.com):
1. From the [console](https://console.crusoecloud.com), click **Admin** in the bottom-left corner.
2. Select **Security** > **[Intelligence API keys](https://console.crusoecloud.com/security/inference-api-keys)** from the left navigation.
3. Click **Create**.
4. (Optional) Enter an alias for your key.
5. (Optional) Enter an expiration date for your key.
6. Copy the **API key**. Make sure that you save the key in a secure location before leaving the page.
### Authenticate against the API
Use your Intelligence API key to authenticate across your intelligence API calls and self-serve deployment management API calls.
1. Export the token and base URL in your shell:
```shell
export API_TOKEN=''
export INFERENCE_URL='https://api.inference.crusoecloud.com/v1/chat/completions'
export DEPLOYMENT_URL='https://api.crusoecloud.com/v1/projects/{project_id}/foundry/selfserve/'
```
2. For inference requests, construct an OpenAI client pointed at the Crusoe gateway. Every Python example on this page assumes you have this `openai_client` in scope:
```python
from openai import OpenAI
import httpx, os
openai_client = OpenAI(
api_key=os.environ["API_TOKEN"],
url=f"os.environ['INFERENCE_URL']",
http_client=httpx.Client(proxy=None, trust_env=False),
)
```
The full OpenAPI specification is published at [api.intelligence.crusoecloud.com/docs](https://api.intelligence.crusoecloud.com/docs).
## 3. Choose a base model and deployment configuration
Define your deployment by selecting the base model you want to serve and the deployment configuration you want to optimize for.
### Deployment configurations
Each deployment offers one or more optimization profiles. Pick the configuration that matches your workload requirements and Crusoe will apply the corresponding engine, hardware, and optimizations for you. There's no hand-tuning required.
| Configuration | Optimization | Best for |
| ------------------ | ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| **Responsiveness** | Low latency, optimized for time-to-first-token | Interactive applications, real-time inference, latency-sensitive workloads |
| **Throughput** | Cost efficiency at scale, optimized for token volume | Batch processing, high-volume workflows, cost-per-token minimization |
| **Balanced** | Hybrid blend of throughput and responsiveness, optimized to support moderate token volume and latency | General purpose production traffic |
### Supported models
Refer to [Available models](/self-serve-deployments/available-models) for the full list of supported base models, which you can also deploy with LoRA adapters trained through [Serverless Fine-Tuning](/serverless-fine-tuning/overview).
## 4. Create a deployment
Create a deployment from the console or API to get started.
**UI:**
1. Sign in to the [console](https://console.crusoecloud.com/) and switch to the **Intelligence Foundry** app in the bottom-left corner.
2. Select **[Self-Serve Deployments](https://console.crusoecloud.com/foundry/deployments)** from the **Inference** section of the left navigation. The page lists every deployment in your project with its status, model, hardware, replicas, and metadata.
3. Click **Create deployment** and complete the form:
- Select a base model and, optionally, a fine-tuned checkpoint
- Select a deployment configuration (Responsiveness, Throughput, or Balanced)
- Select the number of replicas you want your deployment to support
The hourly cost associated with your chosen deployment configuration is displayed before you confirm.
**cURL:**
```bash
curl "$DEPLOYMENT_URL/deployments" \
-X POST \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"flavor_id": "",
"deployment_name": "",
"replicas":
}'
```
After the deployment creation is initiated, your new deployment will appear in the self-serve deployment table. Click any row in the self-serve deployment table to view activity logs, endpoint metadata, and its current replica count.
## 5. Check deployment status
A new deployment provisions reserved capacity, which can take up to 40 minutes to complete. A deployment moves through the following states during its lifecycle:
| State | Description |
| -------------- | ------------------------------------------------------------ |
| `Creating` | Capacity is being provisioned and the engine is starting up. |
| `Ready` | The deployment is ready to serve traffic. |
| `Scaling up` | The deployment is scaling up its active replicas. |
| `Scaling down` | The deployment is scaling down its active replicas. |
| `Syncing` | The deployment alias is being updated. |
| `Failed` | The deployment couldn't be created or updated. |
| `Deleting` | The deployment is being torn down. |
| `Deleted` | The deployment has been removed. |
Check the status of your deployment from the deployment table on the **Self-Serve Deployments** page or directly through the API. The status updates to `Ready` when the deployment is available to serve traffic.
```bash
curl "$DEPLOYMENT_URL/deployments" \
-X GET \
-H "Authorization: Bearer $API_TOKEN" \
```
## 6. Run inference
When the deployment is in `Ready` state, send requests to it through the OpenAI-compatible Chat Completions API using your API key. Pass the deployment alias as the `model`.
**python:**
```python
import os
from openai import OpenAI
client = OpenAI(
base_url=os.environ['INFERENCE_URL'],
api_key=os.environ['API_TOKEN'],
)
response = client.chat.completions.create(
model='',
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Summarize the theory of relativity in one sentence."}
],
)
print(response.to_json())
```
**typescript:**
```typescript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: process.env.INFERENCE_URL,
apiKey: process.env.API_TOKEN,
});
client.chat.completions
.create({
model: "",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{
role: "user",
content: "Summarize the theory of relativity in one sentence.",
},
],
})
.then((response) => console.log(response));
```
**cURL:**
```shell
curl $INFERENCE_URL \
--request 'POST' \
--header 'Content-Type: application/json' \
--header 'Accept: text/event-stream' \
--header "Authorization: Bearer $API_TOKEN" \
--data '{
"model": "",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Summarize the theory of relativity in one sentence."}
]
}'
```
Because your deployment runs on reserved capacity, its throughput is bounded by replica count rather than a shared rate limit. To add headroom for traffic spikes, increase the replica count on the deployment in the next step.
## 7. Manage deployments
You can update a running deployment (for example, to change replica count or the deployment alias) or delete one you no longer need. When you delete a deployment, billing stops.
To view all deployment management options, select the three-dot icon on any deployment row on the **Self-Serve Deployments** page. The menu exposes options to edit the deployment alias, update the replica count, or delete the deployment.
### Configure notifications
By default, [notifications](/managed-ai/notifications) are sent when a self-serve deployment is created, deleted, or scaled. You can manage your preferences from the console's [Notifications settings](https://console.crusoecloud.com/notifications) page.
### Edit a deployment alias
To update the alias for a deployment:
**UI:**
1. Define a unique name for your deployment.
2. Confirm your new deployment name.
**cURL:**
```bash
curl "$DEPLOYMENT_URL/deployments/{id}" \
-X PATCH \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"deployment_name": ""
}'
```
The deployment status updates to `Syncing` while the alias is updated, and returns to `Ready` when the update is complete.
### Update replica counts
To adjust the number of replicas for a deployment:
**UI:**
1. Select a value that reflects your expected traffic load and fits within your allotted quota.
2. Confirm your new replica count.
**cURL:**
```bash
curl "$DEPLOYMENT_URL/deployments/{id}" \
-X PATCH \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"replicas": ""
}'
```
The deployment status updates to `Scaling up` or `Scaling down` based on the direction of the change, and returns to `Ready` when the update is complete. The deployment can still serve traffic while the replica count is being adjusted.
### Delete a deployment
After you confirm deletion, the deployment status updates to `Deleting`, and the deployment disappears from the list once the deletion is complete. Billing stops when the deployment is deleted.
```bash
curl "$DEPLOYMENT_URL/deployments/{id}" \
-X DELETE \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
```
### View deployment details
To view the deployment overview, activity log, metadata, and sample code for inferencing, select the deployment alias from the **Self-Serve Deployments** page or use the API.
```bash
curl "$DEPLOYMENT_URL/deployments/{id}" \
-X GET \
-H "Authorization: Bearer $API_TOKEN"
```
## 8. (Optional) Deploy a fine-tuned model
Self-serve works with Crusoe serverless fine-tuning to help you get your tuned models to production with one click. A LoRA adapter you train there is registered in the same model registry as the base models, so it appears in the models list as soon as training completes.
You have three ways to deploy a fine-tuned checkpoint:
- **From the Self-Serve Deployments page:** Follow the [Create a deployment](#4-create-a-deployment) steps, then select your fine-tuned checkpoint from the list after you select the corresponding base model architecture.
- **From a Fine-tuned model's [Jobs](https://console.crusoecloud.com/foundry/fine-tuning/jobs) page:** Click the three-dot menu next to the checkpoint you want to deploy and select **Deploy**.
- **From the self-serve deployments API:** First, retrieve your `fine_tuned_model` identifier for your desired fine-tuned model checkpoint and include that in your self-serve deployment creation request.
```bash
curl "$DEPLOYMENT_URL/deployments" \
-X POST \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"fine_tuned_model_id": "",
"flavor_id": "",
"deployment_name": "",
"replicas":
}'
```
Because fine-tuning and deployment share the same registry and API conventions, you can iterate quickly: train a new adapter, point a new deployment at it, and shift traffic—without rebuilding your serving stack.
## Next steps
- Need optimization beyond the standard configurations? [Contact us](https://www.crusoe.ai/contact-sales) about Tailored Deployments
- For an overview of Crusoe's Managed AI options, see [Managed AI](/managed-ai/overview)
---
# Serverless Fine-Tuning
With [Serverless Fine-Tuning](/serverless-fine-tuning/overview), you can fine-tune open models using a LoRA-based supervised training workflow through the Intelligence Foundry. You upload a dataset, launch a job, monitor training, and deploy the custom model.
Follow this quickstart to fine-tune a base model for your use case or to test how the [Serverless Fine-Tuning API](/api/managed-ai/) works. For a more detailed walkthrough, see [Fine-tune a model](/serverless-fine-tuning/fine-tune-a-model).
## Prerequisites
- A training dataset in JSON Lines (JSONL) or Parquet format, 3 GB or smaller.
- Python 3.9 or later (if you use the Python client), or curl.
## 1. Log in or create an account
Log in to the [Crusoe Cloud Console](https://console.crusoecloud.com) or [Create an account](/create-an-account).
After you log in, switch to the **Intelligence Foundry** app in the bottom-left of the [console](https://console.crusoecloud.com/).
## 2. Generate an API key and authenticate
To create an API key through the [console](https://console.crusoecloud.com):
1. From the [console](https://console.crusoecloud.com), click **Admin** in the bottom-left corner.
2. Select **Security** > **[Intelligence API keys](https://console.crusoecloud.com/security/inference-api-keys)** from the left navigation.
3. Click **Create**.
4. (Optional) Enter an alias for your key.
5. (Optional) Enter an expiration date for your key.
6. Copy the **API key**. Make sure that you save the key in a secure location before leaving the page.
### Authenticate against the API
All API calls require an Intelligence API key and the API base URL.
1. Export the token and base URL in your shell:
```shell
export API_TOKEN=''
export URL='https://api.intelligence.crusoecloud.com'
```
2. Construct an OpenAI client pointed at the Crusoe gateway. Every Python example on this page assumes you have this `client` in scope:
```python
from openai import OpenAI
import httpx, os
client = OpenAI(
api_key=os.environ["API_TOKEN"],
base_url=f"{os.environ['URL']}/v1",
http_client=httpx.Client(proxy=None, trust_env=False),
)
```
For more information, see [Manage API keys](/identity-and-security/managing-api-keys).
## 3. Upload a training dataset
The example below uses the `openai` Python client because the [Serverless Fine-Tuning API](/api/managed-ai/) is OpenAI-compatible. Substitute your dataset path for `train.jsonl`.
1. Install the OpenAI Python client:
```shell
pip install openai httpx
```
2. Create a client pointed at the Crusoe gateway and [upload the file](/api/managed-ai/#tag/Files/operation/createFile):
```python
from openai import OpenAI
import httpx, os
client = OpenAI(
api_key=os.environ["API_TOKEN"],
base_url=f"{os.environ['URL']}/v1",
http_client=httpx.Client(proxy=None, trust_env=False),
)
with open("train.jsonl", "rb") as f:
file_obj = client.files.create(file=f, purpose="fine-tune")
print(file_obj.id)
```
3. Save the printed file ID. You'll pass it to the job creation call.
## 4. Launch a fine-tuning job
1. [Create a job](/api/managed-ai/#tag/Fine-tuning/operation/createFineTuningJob) that references your uploaded file and a base model. Substitute the file ID and a supported base model ID:
```python
job = client.fine_tuning.jobs.create(
model=BASE_MODEL_ID,
training_file=train_file.id,
validation_file=val_file.id,
suffix="custom-model",
method={
"type": "supervised",
"supervised": {
"hyperparameters": {
"n_epochs": 3,
"batch_size": 64,
"learning_rate": 0.0001,
},
},
},
)
print(job.id)
```
2. Use [`retrieve`](/api/managed-ai/#tag/Fine-tuning/operation/retrieveFineTuningJob) to poll the job until it succeeds:
```python
retrieved = client.fine_tuning.jobs.retrieve(job.id)
print(retrieved.status)
```
You can also watch live training loss, validation loss, and the estimated time to completion (ETA) on the **[Fine-Tuning page](https://console.crusoecloud.com/foundry/fine-tuning/jobs)** in the Intelligence Foundry.
## 5. (Optional) Download a checkpoint
1. [List the checkpoints](/api/managed-ai/#tag/Fine-tuning/operation/listFineTuningJobCheckpoints) emitted by the finished job:
```python
checkpoints = client.fine_tuning.jobs.checkpoints.list(job.id).data
for ckpt in checkpoints:
print(ckpt.id, ckpt.metrics)
```
2. [Download the checkpoint](/api/managed-ai/#tag/Files/operation/downloadFile) you want. The example uses curl for the binary stream:
```shell
BEST_CHECKPOINT=''
curl -s $URL/v1/files/${BEST_CHECKPOINT}/content \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN" \
--output checkpoint-best.zip
```
3. Unzip the archive on the machine you'll serve from.
## 6. Self-serve a deployment for your checkpoint
From the fine-tuned model's job details page, find the checkpoint you want to deploy, click the corresponding three-dot menu, and select **Deploy**. For more information on self-serve deployments, see [Self-Serve Deployments](/serverless-fine-tuning/overview).
---
# Installing and configuring the CLI
# Getting Started with the Crusoe CLI
## Installing and Configuring the CLI
### Step 1: Install
You can install the Crusoe CLI via a number of common package managers, including `brew` and `apt`.
**Mac OS:**
On Mac, use [`homebrew`](https://brew.sh) to install the Crusoe CLI:
```sh
brew install crusoecloud/cli/crusoe
```
To upgrade to a newer version of the CLI, use `brew upgrade crusoe`.
**Linux:**
On Linux, use `apt` to install the Crusoe CLI:
```sh
echo "deb [trusted=yes] https://apt.fury.io/crusoe/ * *" > /etc/apt/sources.list.d/fury.list
sudo apt update
sudo apt install crusoe
```
To upgrade to a newer version of the CLI, use `sudo apt upgrade crusoe`.
Otherwise, to use `yum`, visit the [latest GitHub release](https://github.com/crusoecloud/cli/releases/latest) and download the `.rpm` asset suited for your machine, and run `sudo yum install `.
**Windows:**
On Windows, visit the [latest GitHub release](https://github.com/crusoecloud/cli/releases/latest) and download the `crusoe_Windows_$PLATFORM.tar.gz` asset suited for your machine (e.g. `crusoe_Windows_arm64.tar.gz`), untar it, and add the binary to your system PATH.
### Step 2: Get your API Keys from the Console
To authenticate with the Crusoe CLI, you will need an API access key ID and a secret key. You can generate these in the Crusoe Cloud console from the [Security page](https://console.crusoecloud.com/security/tokens).
:::warning
**Warning:** You will only be able to view the Secret key once! Ensure that you save it somewhere secure before you refresh or leave the page.
:::
### Step 3: Configure the CLI with Defaults for your Account
Run the following command to set up defaults for the CLI to use. It will ask for the API keys saved above, and a project, which can be found on your [Projects page](https://console.crusoecloud.com/projects). Under the hood, this will create and populate a `default` profile in `~/.crusoe/config`.
```sh
crusoe config init
```
To see current values for your default profile, use `crusoe config get `. To update them, use `crusoe config set `. The variables are `default_project`, `access_key_id`, `secret_key`, and optionally, `ssh_public_key_file`.
### Step 4: Test the CLI
If you've properly installed and configured the CLI, you should be able to run the `crusoe whoami` command and see the logged-in user:
```sh
> crusoe whoami
user@domain.com
```
You're ready to go! Now, try [creating a VM](/quickstart/creating-a-vm).
## Alternative Ways to Configure the CLI
### Using the Config File Directly and Switching Between Multiple Projects
You can edit your config directly to work with multiple profiles. To validate your changes, run:
```sh
$ cat ~/.crusoe/config
[default]
default_project=""
access_key_id=""
secret_key=""
ssh_public_key_file=
[my-other-environment]
default_project=""
access_key_id=""
secret_key=""
ssh_public_key_file=
```
To switch between multiple profiles in your `~/.crusoe/config` file, use the environment variable `CRUSOE_PROFILE` (e.g., `export CRUSOE_PROFILE="my-other-environment"`).
### Use Environment Variables
To use the CLI without a config file, or to override its values, set analogous environment variables:
- `CRUSOE_DEFAULT_PROJECT`
- `CRUSOE_ACCESS_KEY_ID`
- `CRUSOE_SECRET_KEY`
## Conclusion
You can now explore all of the easy-to-use CLI commands for managing your AI infrastructure. Use `crusoe --help` to get started. Now, try [creating a VM](/quickstart/creating-a-vm) with the CLI.
If you're having issues, check that you've properly installed the CLI, have a config file locally at `~/.crusoe/config`, and see a default profile with your API keys. If this still isn't working, [contact support](/resources/support).
---
# Getting Started with Terraform
## Installing and Configuring Terraform for Crusoe Cloud
This Quick Start Guide will walk you through the process of installing and configuring Terraform for use with Crusoe Cloud.
### Step 1: Prerequisites
Ensure that your system meets the following prerequisites before proceeding:
- Operating System: Compatible with Windows, macOS, or Linux.
- Internet Connection: Required for downloading Terraform and related dependencies.
- Crusoe Account: Ensure you have an active Crusoe account to access resources.
### Step 2: Download Terraform
1. Visit the official [Terraform website](https://www.terraform.io/)
2. Navigate to the "Downloads" section and choose the appropriate version for your operating system.
3. Follow the installation instructions provided on the Terraform website to complete the installation.
### Step 3: Verify Installation
4. Open a terminal or command prompt.
5. Run the following command to verify the successful installation:
```sh
terraform --version
```
This should display the installed Terraform version, confirming a successful installation.
### Step 4: Authenticate with Crusoe
6. Terraform looks for a config file at `~/.crusoe/config` which contains credentials as well as any defaults.
7. Please follow the steps to create API keys by heading to [Managing API keys](../identity-and-security/managing-api-keys.mdx)
8. At a minimum, the following options are required to authenticate to the Crusoe API:
```sh
[default]
default_project="default"
access_key_id=""
secret_key=""
```
Replace `default_project` with your actual project name, `access_key_id` and `secret_key` with your actual access key and secret key.
### Step 5: Configure Terraform for Crusoe
9. Open your terminal or command prompt.
10. Navigate to the directory where you want to store your Terraform configurations.
11. Create a new file named `main.tf` in the same directory and add the following to your Terraform configuration code.
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
```
12. Run the following command to initialize Terraform in the directory: `terraform init`
## Conclusion
You have successfully installed and configured Terraform for use with Crusoe Cloud! Now, you can start creating resources. For example, visit [Manage your VMS](https://docs.crusoecloud.com/compute/virtual-machines/managing-vms/index.html) and click on the “Terraform” tab to create a VM. If you run into any issues, [contact support](../resources/support.md).
---
# Creating a VM
## Creating and uploading an SSH key
If you don't already have an SSH key, you will need to create one on your local machine.
When creating a VM for the first time, you will be prompted to upload an SSH key, which the VM will use to authenticate future SSH attempts. It will be saved into your account with the name "Default" and accessible on subsequent VM creations from the UI. You can manage all SSH keys in the Console's [Security page](https://console.crusoecloud.com/security/keys).
:::info
Crusoe Cloud supports all SSH public key formats that are accepted by OpenSSH. These include:
- sk-ecdsa-sha2-nistp256@openssh.com
- ecdsa-sha2-nistp256
- ecdsa-sha2-nistp384
- ecdsa-sha2-nistp521
- sk-ssh-ed25519@openssh.com
- ssh-ed25519
- ssh-dss
- ssh-rsa
For more information on the authorized key format please see the [OpenSSH docs](https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT).
:::
## Creating a new VM
**CLI:**
Use the `compute vms create` command to create a VM of your choice. As an example, you can create a VM that uses Nvidia H100 GPUs:
```sh
crusoe compute vms create \
--name my-vm \
--type h100-80gb-sxm-ib.8x \
--location us-southcentral1-a \
--image ubuntu22.04:latest \
--keyfile ~/.ssh/id_ed25519.pub
```
You can find possible values for type and location by running `crusoe compute vms types` and `crusoe locations list` respectively.
If you don't specify an image, the VM will default to the latest version of ubuntu 22.04. Run `crusoe compute images list` for more options.
**UI:**
To create a VM via the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Click **Create Instance**.
3. Input all required information, including instance type, location, name, and so on.
4. Click **Create Instance**.
**Terraform:**
Creating and accessing VMs is the first step to getting started on Crusoe Cloud. The following is intended to help get you started using Terraform to provision a VM in Crusoe Cloud.
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates a VM that uses Nvidia H100 GPUs called “my-vm”:
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
// local files
locals {
ssh_key = file("~/.ssh/id_ed25519.pub") # replace with path to your public SSH key if different
}
// new VM
resource "crusoe_compute_instance" "my_vm" {
name = "my-vm"
type = "h100-80gb-sxm-ib.8x"
location = "us-southcentral1-a"
image = "ubuntu22.04:latest" # use the 'latest' flag to get the most up-to-date image available from Crusoe
ssh_key = local.ssh_key
}
```
`name`, `type`, `location` and `ssh_key` are required arguments. `default_project` is also required, but is typically specified in the config file, which is why it's omitted in the `crusoe_compute_instance` resource (see step 4 above). `image` is not required and for this example, we will use an `ubuntu22.04:latest` image.
After saving the code to a `main.tf` file, the following commands serve as the process to create a resource in Crusoe Cloud using Terraform:
`terraform init` - Initializes a working directory containing Terraform configuration files.
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will create the resources.
You can confirm that terraform successfully created the resources through the console, or via the CLI with `crusoe compute vms list`.
## Access the VM via SSH
If VM creation succeeds, the VM will start and you will be provided with an IP address for you to SSH into:
```sh
ssh ubuntu@ # or root@ (deprecated)
```
For more details on VMs, see the [Compute section](../compute).
---
# Spin up your first GPU cluster
Provision your first multi-node GPU cluster using Crusoe Managed Kubernetes (CMK) or Slurm. When your cluster is up, you can use it to train models, serve inference, and run data processing pipelines.
**Select an orchestration path:**
**cmk:**
Crusoe Managed Kubernetes (CMK) gives you a Kubernetes control plane with GPU drivers, network operators, and storage add-ons preconfigured—so you can go from a fresh account to a distributed training job quickly.
## Prerequisites
- A Crusoe Cloud account with an active project. See [Create an account](/create-an-account)
- The [Crusoe CLI](/installing-the-cli) installed and authenticated
- An SSH public key registered on your account. See [Manage your SSH keys](/compute/virtual-machines/managing-ssh-keys)
- [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed locally
## 1. Create a cluster
You can create clusters using the `kubernetes clusters create` command. Use the `-help` flag for an exhaustive list of options.
```sh
crusoe kubernetes clusters create \
--name my-first-cluster \
--cluster-version 1.31 \
--location us-east1-a \
--subnet-id 6f8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c
--add-ons "nvidia_gpu_operator,nvidia_network_operator,crusoe_csi"
```
You may list the Kubernetes versions available for cluster creation by using the 'kubernetes clusters list-versions' command. Specifying an unqualified version (e.g. 1.30) when creating a cluster will provision the latest stable patch version associated with the minor version.
To create a CMK cluster via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Click **Create Cluster**.
3. Follow the UI flow to input all required elements.
4. Optional selections include specifying the Service and Pod network CIDRs for Cilium and selecting one or more add-ons to deploy into the cluster.
5. Click **Create**.
## 2. Add a GPU node pool
Node pools hold the actual GPU workers. Attach one to your cluster.
Nodepools can be created by using the `kubernetes nodepools create` command. Nodepools must be created in the context of a specific cluster. Use the '--help' flag for an exhaustive list of options.
```sh
crusoe kubernetes nodepools create \
--name my-first-nodepool \
--cluster-id 6f8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c \
--type h100-80gb-sxm-ib.8x \
--count 4 \
--ib-partition-id 4c8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c \
```
To create a node pool via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Select the cluster you want to edit.
3. Click **Create Node Pool**.
4. Fill out the required fields specifying the type of nodes you want to create and the count.
5. Click **Create**.
## 3. Connect
Retrieve credentials for your cluster:
Use the `kubernetes clusters get-credentials` command to retrieve credentials for a specific cluster.
```sh
crusoe kubernetes clusters get-credentials
```
By default, credentials are stored in a file named `~/.kube/config`. You may alter the path credentials are stored at by using the `--kubeconfig-path` flag. If you have existing configs stored in the same path, the new cluster kubeconfig will be appended to the end and set as the current context.
If you are an admin user, you can retrieve your cluster admin kubeconfig via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Select the cluster you want credentials for.
3. Click **Generate Kubeconfig** in the top right.
4. Your kubeconfig will be downloaded.
Then confirm your nodes came up:
```shell
kubectl get nodes
```
All nodes should show `STATUS: Ready` within a couple of minutes. If a node stays `NotReady`, describe it with `kubectl describe node `—usually the GPU operator is still installing drivers.
:::tip
Prefer to manage your cluster as code with Terraform? Use the [Manage your CMK clusters](/orchestration/cmk/managing-clusters) and [Manage your Node Pools](/orchestration/cmk/managing-nodepools) guides.
:::
**slurm:**
Slurm runs on top of Crusoe Managed Kubernetes (CMK). With this, you get the familiar Slurm scheduler UX (`sbatch`, `srun`, `squeue`, `sinfo`) with a Crusoe-managed Kubernetes control plane underneath, plus the same GPU hardware, InfiniBand fabric, and operator add-ons as CMK. If you already have an SSH key registered on your account, cluster creation is done with one-click. If you don't have an SSH key registered, you can supply a key during creation (and optionally adjust the VPC or subnet). The CLI mirrors that flow with a single command.
See the [Slurm quickstart](/orchestration/slurm/quickstart) for the Terraform flow and additional details, including the full flag reference.
## Prerequisites
- A Crusoe Cloud account with an active project. See [Create an account](/create-an-account).
- The [Crusoe CLI](/installing-the-cli) installed and authenticated.
- An SSH public key registered on your account. See [Manage your SSH keys](/compute/virtual-machines/managing-ssh-keys).
## 1. Create a Slurm cluster
Create a Managed Slurm cluster with a single command. This provisions the underlying Kubernetes cluster, Slurm controller, login nodes, and shared storage, and installs the required add-ons automatically.
```shell
crusoe slurm clusters create \
--name my-slurm-cluster \
--location us-southcentral1-a \
--keyfile ~/.ssh/id_ed25519.pub
```
The command waits for the operation to complete and prints the result.
To create a Managed Slurm cluster via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Slurm](https://console.crusoecloud.com/orchestration/slurm)** in the left nav.
2. Click **Create Cluster**, enter a **Public SSH Key**, and fill the remaining required fields.
3. Click **Create**.
## 2. Add worker nodes
Worker capacity is provided by node sets attached to the cluster. You can create GPU (NVIDIA) node sets for training and inference workloads, or CPU-only node sets for preprocessing, orchestration, or other auxiliary workloads.
```shell
crusoe slurm nodesets create \
--name gpu-workers \
--cluster-name my-slurm-cluster \
--type b200-180gb-sxm-ib.8x \
--count 2 \
--ib-partition-id
```
Use `--ib-partition-id` for multi-node InfiniBand workloads. See [Supported GPU Types](/orchestration/slurm/overview#supported-gpu-types) for available instance types.
1. From your Slurm cluster's page in the console, click **+ Create Nodeset**.
2. In the **Create Nodeset** modal, accept the auto-generated name or enter your own.
3. Choose the workload type: **NVIDIA** or **CPU Only**.
4. Select an **Instance Type** and fill out the remaining required fields.
5. Click **Create Nodeset**.
## 3. Connect and submit a job
Grab the login-node endpoint from `crusoe slurm clusters get `, then SSH into the cluster and confirm the scheduler is up:
```shell
ssh root@
sinfo
# Worker nodes should show as idle and ready.
srun --gpus=8 nvidia-smi
```
If you are an admin user, you can retrieve your cluster admin kubeconfig via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Slurm](https://console.crusoecloud.com/orchestration/slurm)** in the left nav.
2. Select the cluster you want credentials for.
3. Click **Download Kubeconfig** in the top right.
## Next steps
Decide how you want to use your cluster:
- **Train models**—Run distributed training on B200 GPUs with InfiniBand interconnects, using frameworks like PyTorch DDP, NVIDIA NeMo, DeepSpeed, and Ray Train. Multi-node jobs scale across the cluster's IB fabric without additional configuration on your end.
- **Serve models**—Run vLLM, NVIDIA Triton, or your own inference stack on hardware you control, colocated with your data and free from third-party rate limits. Autoscale replicas with the Cluster Autoscaler on CMK, or fix capacity with a dedicated Slurm partition.
- **Run data processing**—Execute GPU-accelerated preprocessing, embedding generation, or synthetic data pipelines with RAPIDS, Ray Data, or Spark on GPU. Reuse the same cluster for training and preprocessing to keep data close to the compute.
---
# Managed AI
Crusoe gives you fully managed options for [training](/serverless-fine-tuning/overview), [running](/serverless-inference/overview), and [hosting](/self-serve-deployments/overview) AI models—without provisioning or operating GPU infrastructure yourself.
- [Serverless Inference](/serverless-inference/overview): Run inference workloads with Crusoe's Managed AI services.
- [Self-Serve Deployments](/self-serve-deployments/overview): Spin up dedicated, self-serve inference deployments on managed infrastructure.
- [Serverless Fine-Tuning](/serverless-fine-tuning/overview): Fine-tune an open model using a LoRA-based supervised training workflow through the Intelligence Foundry.
---
# Usage and billing
You can view usage and billing information through [dedicated views](https://console.crusoecloud.com/foundry/usage) on the Intelligence Foundry for the services in the following table:
| Offering | Billing |
| -------------------------------------------------------------- | --------------------- |
| **[Serverless Inference](/serverless-inference/overview)** | Per-token |
| **[Serverless Fine-Tuning](/serverless-fine-tuning/overview)** | Per-token |
| **[Self-Serve Deployments](/self-serve-deployments/overview)** | GPU-hour (time-based) |
:::info
For pricing information, refer to [https://www.crusoe.ai/cloud/pricing](https://www.crusoe.ai/cloud/pricing).
:::
**UI:**
To view usage and billing information from the console:
1. Log in to the [Crusoe Cloud console](https://console.crusoecloud.com).
2. Navigate to the [Intelligence Foundry](https://console.crusoecloud.com/foundry) app in the bottom-left corner.
3. Click **Usage** in the left navigation.
4. Select an option to view its usage information.
5. To view billing information, click **Billing** in the left navigation and select an option.
---
# Serverless Fine-Tuning
Train open models for your domain or task—without provisioning or managing GPU infrastructure yourself—using serverless fine-tuning. After you evaluate the training results, use [self-serve deployments](/self-serve-deployments/overview) to deploy your fine-tuned model to a dedicated inference endpoint.
- [How it works](/serverless-fine-tuning/how-it-works): Concepts, lifecycle, and architecture behind serverless fine-tuning.
- [Quickstart](/serverless-fine-tuning/quickstart): Run your first fine-tuning job end-to-end in a few minutes.
- [Fine-tune a model](/serverless-fine-tuning/fine-tune-a-model): Configure datasets, hyperparameters, and checkpoints for a production run.
- [Available models](/serverless-fine-tuning/available-models): Explore the open models available for fine-tuning.
---
# How Serverless Fine-Tuning works
Serverless fine-tuning adapts open models to your domain or task using LoRA-based supervised training, without provisioning or managing GPU infrastructure yourself. You provide a dataset, configure a job, and Crusoe handles scheduling, training, and checkpoint storage.
## The fine-tuning lifecycle
Use the diagram below to understand the fine-tuning lifecycle and which steps of the lifecycle you're responsible for:
When the job succeeds, use the checkpoint ID to deploy your custom
model through{" "}
Self-Serve Deployments
.
),
},
{
owner: "You",
title: "(Optional) Download checkpoints",
description:
"When the job succeeds, the job record lists every checkpoint's file ID. Download the one you want and serve it from your own inference stack.",
},
]}
/>
## Key use cases
- **Domain adaptation:** Teach a general-purpose model the vocabulary, tone, and conventions of a specific industry or product area, for example, legal, medical, or customer support
- **Task-specific behavior:** Train a model to follow a strict output format, such as always returning JSON, a classification label, or a structured report
- **Latency and cost reduction:** Distill a large model's behavior into a smaller, faster model for high-throughput or latency-sensitive production workloads
- **Persona and tone alignment:** Fine-tune a model to match a specific brand voice, communication style, or persona consistently across outputs
- **Private data grounding:** Incorporate proprietary knowledge that can't be shared with third-party model providers, keeping sensitive data on your own infrastructure
## Key capabilities
- Fully serverless—with no instance lifecycle to manage. You pay for the job, not idle GPUs.
- Automated job queueing, scheduling, monitoring, and failure recovery
- OpenAI-compatible API surface for files, jobs, and checkpoints, so existing OpenAI fine-tuning code works with minimal changes.
- Automatic checkpointing during training, with downloadable LoRA adapters for each checkpoint.
## What gets trained
Serverless fine-tuning trains a LoRA adapter, not the full base model. The base model's weights stay frozen, and the trainer learns a small set of low-rank matrices that modify the model's behavior. This keeps training fast and produces a compact adapter (tens or hundreds of megabytes) that you load on top of the original base model at inference time.
The trainer applies loss only to assistant turns in each conversation. User and system turns are masked, so the model learns what to produce, not what to consume.
## Datasets and checkpoints
Datasets and checkpoints both live in the [Files API](/api/managed-ai/#tag/Files) and share the same lifecycle primitives:
| **Artifact** | **Created when** | **Format** | **Used for** |
| ---------------------- | ---------------------------------- | ------------------------ | ------------------------------ |
| **Training dataset** | You upload it | JSONL chat conversations | Input to a job |
| **Validation dataset** | You upload it (optional) | JSONL chat conversations | Evaluation during training |
| **Checkpoint** | The trainer emits it automatically | LoRA adapter archive | Downloading and serving models |
If you don't supply a validation dataset, Crusoe holds out 10% of the training data automatically.
Each checkpoint is independently usable. After a run finishes you can compare validation loss across checkpoints and pick the one with the best fit for your task.
## How serverless fine-tuning differs from VM-based training
Three things distinguish serverless fine-tuning from running a training job on a virtual machine (VM):
- **No instance lifecycle:** You don't have to request, monitor, or terminate GPUs. The platform handles provisioning and teardown.
- **No persistent storage to manage:** Datasets and checkpoints live in the Files API. You don't need to attach disks or mount object stores.
- **Per-job billing:** You pay for the training job, not for idle GPUs between jobs.
## API access
The interface is OpenAI-compatible, so you can drive the service with the official `openai` Python client or with `curl` against the REST API.
- **Base URL:** `https://api.intelligence.crusoecloud.com`
- **OpenAPI specification:** `https://api.intelligence.crusoecloud.com/docs`
## Next steps
[Self-serve deployments](/self-serve-deployments/overview): Run inference on your model after a fine-tuning job completes by deploying it with self-serve deployments.
---
# Serverless Fine-Tuning quickstart
---
# Fine-tune a model
Serverless fine-tuning lets you fine-tune open-weight models on your own data without provisioning or managing GPU infrastructure. You upload a training dataset, choose a base model, and configure hyperparameters. Crusoe handles the rest and returns LoRA adapter checkpoints you can serve from your own inference stack.
For API-specific information, refer to the [Serverless Fine-Tuning API](/api/managed-ai/).
## Prerequisites
- An Intelligence API key generated from the **[API Keys](https://console.crusoecloud.com/security/inference-api-keys)** page in Intelligence Foundry. See [Managing API Keys](/identity-and-security/managing-api-keys) for more information.
- The OpenAI Python client (`pip install openai httpx`), if you use the Python examples.
- A training dataset in JSON Lines (JSONL) or Parquet format, 3 GB or smaller.
## 1. Authenticate against the API
All API calls require an Intelligence API key and the API base URL.
1. Export the token and base URL in your shell:
```shell
export API_TOKEN=''
export URL='https://api.intelligence.crusoecloud.com'
```
2. Construct an OpenAI client pointed at the Crusoe gateway. Every Python example on this page assumes you have this `openai_client` in scope:
```python
from openai import OpenAI
import httpx, os
openai_client = OpenAI(
api_key=os.environ["API_TOKEN"],
base_url=f"{os.environ['URL']}/v1",
http_client=httpx.Client(proxy=None, trust_env=False),
)
```
## 2. Prepare a dataset
You must [upload](/api/managed-ai/#tag/Files/operation/createFile) training data as a single file that's 3GB or smaller with a `.jsonl` or `.parquet` extension. Each line must be a JSON object containing a `messages` array of chat turns.
### Dataset format
Each line is a chat conversation. The example below classifies banking customer messages into intent labels:
```jsonl
{
"messages": [
{
"role": "system",
"content": "You are a banking customer service intent classifier. Given a customer message, classify it into exactly one of these intents: activate_my_card, country_support, ..."
},
{
"role": "user",
"content": "Which countries are represented?"
},
{
"role": "assistant",
"content": "country_support"
}
]
}
```
### How loss is computed
Loss is applied to assistant turns and masked on every other role.
:::note Multi-turn reasoning isn't supported yet
Single-turn reasoning and multi-turn non-reasoning both work. Contact support if you need multi-turn reasoning.
:::
### Model-specific dataset quirks
Some base models have quirks that affect how you structure assistant turns:
- **llama-3-instruct:** When an assistant turn contains both a tool call and content, llama-3-instruct doesn't render the content. The model wasn't pretrained to emit content before a tool call, only one or the other. Keep this format in your fine-tuning data; don't expect the model to emit content alongside a tool call.
- **gemma-4:** When an assistant turn contains both a tool call and content, gemma-4 renders the content before the tool call, which often reads unnaturally. To get content after the tool call, split it into a separate assistant turn that follows the tool call.
- **gpt-oss:** gpt-oss is pretrained to emit `analysis` before `final`. If your dataset focuses only on final content, include an empty `thinking` field so the model doesn't fight its pretrained format:
```jsonl
{
"messages": [
{
"role": "system",
"content": "..."
},
{
"role": "user",
"content": "Which countries are represented?"
},
{
"role": "assistant",
"content": "country_support",
"thinking": ""
}
]
}
```
## 3. Upload a dataset
[Upload](/api/managed-ai/#tag/Files/operation/createFile) your prepared `.jsonl` file with `purpose=fine-tune`. The response includes a file ID you'll pass in the job creation step:
**python:**
```python
with open("train.jsonl", "rb") as f:
file_obj = openai_client.files.create(file=f, purpose="fine-tune")
print(file_obj.id) # save this for the job creation step
```
**curl:**
```shell
curl $URL/v1/files \
--request POST \
--header 'Content-Type: multipart/form-data' \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN" \
--form "file=@" \
--form "purpose=fine-tune"
```
### Manage uploaded datasets
After upload, you can [list](/api/managed-ai/#tag/Files/operation/listFiles) every dataset in your account, [download](/api/managed-ai/#tag/Files/operation/downloadFile) a dataset's contents, or [delete](/api/managed-ai/#tag/Files/operation/deleteFile) a dataset you no longer need. The examples below show all three operations:
**python:**
```python
# List
files = openai_client.files.list()
# Download
file_content = openai_client.files.content(file_id)
# Delete
openai_client.files.delete(file_obj.id)
```
**curl:**
```shell
# List
curl $URL/v1/files \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN"
# Download
curl $URL/v1/files/${FILE_ID}/content \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN"
# Delete
curl $URL/v1/files/${FILE_ID} \
--request DELETE \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN"
```
## 4. Launch a fine-tuning job
A job requires a training dataset and accepts an optional validation dataset. If you don't supply validation data, Crusoe holds out 10% of the training data automatically.
[Create a job](/api/managed-ai/#tag/Fine-tuning/operation/createFineTuningJob) from either the console or the API. Select the tab for your preferred interface.
**console:**
1. Sign in to the [Crusoe Console](https://console.crusoecloud.com/) and switch to the **Intelligence Foundry** app in the bottom-left of the console.
2. Select **Model Shaping > [Serverless Fine-Tuning](https://console.crusoecloud.com/foundry/fine-tuning/jobs)** in the left navigation. From this page, you can view all jobs in your project with their current status, base model, and checkpoint count.
3. Click **Start a fine-tuning job** and complete the form by adding a base model, training data, optional validation data, and hyperparameters. If you set a [Customer-managed encryption key](/identity-and-security/customer-managed-keys) in your job's project, all future jobs in that project will automatically use the CMEK for encryption and decryption.
4. Click any row on the **Fine-Tuning** page to see live training and validation loss, ETA, and the list of saved checkpoints.
**api:**
```python
job = client.fine_tuning.jobs.create(
model=BASE_MODEL_ID,
training_file=train_file.id,
validation_file=val_file.id,
suffix="custom-model",
method={
"type": "supervised",
"supervised": {
"hyperparameters": {
"n_epochs": 3,
"batch_size": 64,
"learning_rate": 0.0001,
},
},
},
)
print(job.id)
```
```shell
curl $URL/v1/fine_tuning/jobs \
--request POST \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN" \
--data '{
"model": "'"$BASE_MODEL_ID"'",
"training_file": "",
"validation_file": "",
"suffix": "dx-demo",
"method": {
"type": "supervised",
"supervised": {
"hyperparameters": {
"n_epochs": 3,
"batch_size": 64,
"learning_rate": 0.0001
}
}
}
}'
```
### Hyperparameter reference
The [job creation API](/api/managed-ai/#tag/Fine-tuning/operation/createFineTuningJob) exposes more training configuration than the console. The table below explains the parameters that aren't self-explanatory:
| Parameter | Default | What it does |
| -------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `learning_rate_multiplier` | `1` | Scales the learning rate. Present mainly for OpenAI compatibility. Unless you need to scale the configured learning rate, leave it empty or set to `1`. |
| `checkpoint_steps` | `100` | How often a checkpoint is saved. Each checkpoint is a usable LoRA adapter, and the best one can be selected after the run. If you use more frequent checkpoints, the job takes longer because each save takes time. |
| `eval_steps_per_epoch` | `4` | The number of evaluation passes per epoch. Higher values produce finer-grained loss curves but can slow the job significantly when the eval dataset is large. |
| `early_stopping_patience` | `null` | The number of evaluation calls without improvement before training stops. Set to `null` (default) to disable early stopping. |
| `lora_variant` | `"lora"` | Defines which LoRA variant to use: `"lora"` or `"rslora"`. |
| `lr_scheduler` | `"cosine"` | How learning rate decays over training: `"cosine"` (smooth decay to zero), `"linear"` (straight-line decay), `"constant"` (no decay), or `"constant_with_warmup"`. |
| `warmup_ratio` | `0.0` | Fraction of training steps where learning rate linearly ramps from 0 to the target. Prevents early instability when gradients are noisy; typical values are 0.03 to 0.1. |
| `overlong_row_behavior` | `"error"` | What to do when a sample exceeds the maximum sequence length: `"error"` (fail the job), `"drop"` (skip the sample), or `"exact"` (truncate or pad to fit). |
### Check job status
After you launch a job, [retrieve](/api/managed-ai/#tag/Fine-tuning/operation/retrieveFineTuningJob) its current status by job ID or [list every job](/api/managed-ai/#tag/Fine-tuning/operation/listPaginatedFineTuningJobs) in your project. Use retrieval to poll a specific run; use the list endpoint to find a job ID you don't already have.
**python:**
```python
# Retrieve a single job
retrieved_job = openai_client.fine_tuning.jobs.retrieve(job.id)
print(retrieved_job.status)
# Paginate through previous jobs
page = openai_client.fine_tuning.jobs.list(limit=20, after=after)
for j in page.data:
print(j.id)
```
**curl:**
```shell
# Retrieve a single job
JOB_ID='ftjob-...'
curl $URL/v1/fine_tuning/jobs/${JOB_ID} \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN"
# List jobs
curl "$URL/v1/fine_tuning/jobs?limit=20" \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN"
```
### Interpret a successful job response
When a job succeeds, the response includes a `result_files` array. Each entry is a checkpoint ID you can pass to the download endpoint in the next step. Save an ID from the response before moving on:
```json
{
"id": "ftjob-82d5c693a9c84eb2907309225797e6f3",
"status": "succeeded",
"result_files": [
"adapter:checkpoint-1-40a614f42c3d423ab314bd691691c2c0:e32cddf8-72b6-4ea3-be48-74eece3554fe:..."
]
}
```
## 5. (Optional) Download a checkpoint
When a job finishes, every saved checkpoint is exposed as a downloadable LoRA adapter. You can [list](/api/managed-ai/#tag/Fine-tuning/operation/listFineTuningJobCheckpoints) the checkpoints emitted by a job and [download](/api/managed-ai/#tag/Files/operation/downloadFile) one to disk. See the following examples for reference:
**python:**
```python
# List the checkpoints emitted by a finished job
checkpoints = openai_client.fine_tuning.jobs.checkpoints.list(job.id).data
# Download a checkpoint's contents
content = openai_client.files.content(checkpoints[-1].id)
with open("checkpoint-best.zip", "wb") as f:
f.write(content.read())
```
**curl:**
```shell
BEST_CHECKPOINT=''
curl -s $URL/v1/files/${BEST_CHECKPOINT}/content \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_TOKEN" \
--output checkpoint-best.zip
```
## 6. Self-serve a deployment for your checkpoint
From the console's [Fine-tuning jobs page](https://console.crusoecloud.com/foundry/fine-tuning/jobs), select your model to open its job details page. From the details page, find the checkpoint you want to deploy, click the corresponding three-dot menu, and select **Deploy**. For more information, see [Self-Serve Deployments](/serverless-fine-tuning/overview).
---
# Available models
Refer to the table on this page to see which base models [Serverless Fine-Tuning](/serverless-fine-tuning/fine-tune-a-model) supports.
For each model's pricing information, see [pricing](https://www.crusoe.ai/cloud/pricing#Serverless-Fine-Tuning).
| MODEL | PROVIDER | TYPE | CONTEXT LENGTH | LICENSE | ACCEPTABLE USE POLICY |
| ------------------------------------------------------------------------------------------------------------------ | -------- | -------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) | DeepSeek | instruct | 1M | [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/LICENSE) | |
| [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31B-it) | Google | instruct | 262k | [Apache License 2.0](https://ai.google.dev/gemma/apache_2) | |
| [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | Meta | instruct | — | [Llama 3.1 Community License Agreement](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE) | [Llama 3.1 Acceptable Use Policy](https://www.llama.com/llama3_1/use-policy/) |
| [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) | Meta | instruct | 131k | [Llama 3.3 Community License Agreement](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/LICENSE) | [Llama 3.3 Acceptable Use Policy](https://www.llama.com/llama3_3/use-policy/) |
| [nvidia/nemotron-3.5-lightning-30b-a3b](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) | NVIDIA | instruct | 1M | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | OpenAI | instruct | — | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) | [Acceptable Use Policy](https://huggingface.co/openai/gpt-oss-20b/blob/main/USAGE_POLICY) |
| [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) | OpenAI | instruct | 131k | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) | [Acceptable Use Policy](https://huggingface.co/openai/gpt-oss-120b/blob/main/USAGE_POLICY) |
| [qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-8B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-8B/blob/main/LICENSE) |
| [qwen/Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) | Qwen | instruct | 262k | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE) |
| [qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/LICENSE) |
| [qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/LICENSE) |
| [qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE) |
| [qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE) |
| [qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | Qwen | instruct | 256k | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE) |
| [zai/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) | Z.ai | instruct | 1M | [MIT License](https://huggingface.co/zai-org/GLM-5.2/blob/main/LICENSE) | |
---
# Serverless Inference
Use serverless inference to interact with [supported models](/serverless-inference/available-models) through Crusoe's Intelligence Foundry APIs. Models are served on Crusoe's proprietary inference engine with MemoryAlloy, a cluster-wide memory fabric with cache-aware routing that maximizes cache hits, improving TTFT and throughput.
**Use Serverless Inference when:**
- Traffic is variable or unpredictable
- You're prototyping or in early development
- Per-token pricing is preferable to reserved capacity
---
- [Getting started](/quickstart/getting-started-with-serverless-inference): Retrieve an API key and run your first inference request against a hosted model using the OpenAI SDK.
- [Available models](/serverless-inference/available-models): Browse the base models supported by Serverless Inference on the Intelligence Foundry.
- [Rate limits](/serverless-inference/rate-limits): Understand tokens-per-minute and requests-per-minute limits, 429 and 503 responses, and rate limit headers for Serverless Inference.
- [Inference metrics](/serverless-inference/inference-metrics): Monitor Serverless Inference performance with built-in metrics and a Prometheus-compatible query API.
---
# Getting Started with Serverless Inference
Crusoe's Serverless Inference Service provides OpenAI compatible endpoints for a number of popular open source models. The models are hosted on Crusoe's inference engine with MemoryAlloy, a proprietary cluster-wide memory fabric with cache-aware routing that improves TTFT and throughput.
The instructions below provide steps to start querying models via the OpenAI SDK. All models are accessible via the `api.inference.crusoecloud.com` path.
## Retrieving your Intelligence API token
You can retrieve your Intelligence API token via the [console](https://console.crusoecloud.com/) by following the steps below.
**UI:**
1. Visit the [Intelligence Foundry](https://console.crusoecloud.com/foundry/models) on the console.
2. Select **Inference** from the left nav.
3. Click **Create API Key** to generate an API key.
4. (Optional) Provide an alias and expiration date.
5. Click **Create** to view and save your API key.
## Querying Text models
After retrieving an API key from the Intelligence Foundry, you can use the OpenAI SDK to make requests. The example below uses the `meta-llama/Llama-3.3-70B-Instruct` model.
Need higher rate limits or reserved capacity?
When interacting with a Serverless endpoint, you might receive a `429 Too Many Requests` response due to rate limits. If you need to exceed the default rate limits, you can:
- [Contact us](https://www.crusoe.ai/contact-sales) for a rate limit increase. This is recommended if you expect your initial launch traffic to exceed the default limits.
- If you need reserved inference capacity, use [Self-Serve Deployments](/self-serve-deployments/overview).
- For default values, response codes, and rate limit headers, see [Serverless Rate Limits](/serverless-inference/rate-limits).
```python
import os
from openai import OpenAI
CRUSOE_API_KEY = os.getenv("CRUSOE_API_KEY")
client = OpenAI(
api_key=CRUSOE_API_KEY,
base_url="https://api.inference.crusoecloud.com/v1",
)
completion = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful, concise assistant."},
{"role": "user", "content": "Who is Robinson Crusoe?"},
],
)
print(completion.choices[0].message.content)
```
---
# Serverless rate limits
Serverless Inference runs models on shared, multi-tenant deployments. To keep capacity fair and latency consistent, Crusoe enforces rate limits that cap how many tokens and requests a project can send to a Serverless Inference endpoint each minute. Limits apply per project and per model, and only to traffic handled by Serverless Inference endpoints.
## How rate limits work
Two limits are tracked each minute:
- **Tokens Per Minute (TPM)**: The total input and output tokens your project can process for a given model in a one-minute window.
- **Requests Per Minute (RPM)**: The total requests your project can send to a given model in a one-minute window.
Each limit is enforced independently. When your project exceeds either one, the endpoint returns a `429 Too Many Requests` response.
## Default limits
Your default limits depend on whether your organization has a payment method on file:
| Account status | TPM per model | RPM per model |
| -------------------- | ------------- | ------------- |
| No payment method | 500,000 | 30 |
| Payment method added | 2,000,000 | 600 |
To automatically raise both limits to 2,000,000 TPM and 600 RPM, [add a payment method](https://console.crusoecloud.com/billing/payments). If your product needs higher limits than that before launching, [contact us](https://www.crusoe.ai/contact-sales) to discuss your expected volume.
:::info 503 Service Unavailable
A 2,000,000 TPM limit doesn't guarantee a successful response. Serverless Inference deployments are shared, so you can still receive a `503 Service Unavailable` while a model scales to meet aggregate traffic, even when you're within your limits.
:::
## Responses when a limit is reached
Serverless Inference returns different status codes depending on whether your project exceeded its limit or the shared endpoint is experiencing unusually high load:
| HTTP status | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `429 Too Many Requests` | Your project exceeded its TPM or RPM limit for the model. |
| `503 Service Unavailable` | Your request is within your limits, but the shared endpoint is experiencing unusually high load and can't accept it right now. |
Both `429` and `503` responses are transient, so you can [retry the request with exponential backoff](#retry-with-exponential-backoff) rather than failing immediately. If you're consistently hitting your rate limit or seeing repeated `503`s, [contact us](https://www.crusoe.ai/contact-sales) so we can help raise your limits or advise on capacity planning.
Need higher rate limits or reserved capacity?
- [Contact us](https://www.crusoe.ai/contact-sales) for a rate limit increase. This is recommended if you expect your initial launch traffic to exceed the default limits.
- If you need predictable response rates or reserved inference capacity, use [Self-Serve Deployments](/self-serve-deployments/overview). They run on dedicated hardware with reserved capacity, which gives you predictable performance and often better economics at scale.
## Retry with exponential backoff
Your rate limit resets every minute, and shared-endpoint load fluctuates continuously, so retrying immediately only adds to the contention. Back off exponentially between attempts instead:
1. Wait a short initial delay (for example, 1 second) before your first retry.
2. Double the delay after each subsequent failed attempt (1s, 2s, 4s, 8s, and so on).
3. Add a small amount of random jitter to each delay so that multiple clients don't retry simultaneously.
4. Cap the delay (for example, at 30–60 seconds) and the number of retries, then surface the error if the request still hasn't succeeded.
Because TPM and RPM limits reset every minute, a request that fails with `429` will typically succeed within the next window. `503` responses usually clear even sooner, as soon as the shared endpoint scales to meet demand.
## Rate limit headers
Every Serverless Inference response includes headers that report your current limit and remaining allowance, so you can throttle client-side without waiting for a `429`:
| Header | Example | Description |
| -------------------------------- | --------- | --------------------------------------------------------------- |
| `x-ratelimit-limit-tokens` | `2000000` | Maximum tokens permitted before the TPM limit is exhausted. |
| `x-ratelimit-remaining-tokens` | `1700000` | Remaining tokens permitted before the TPM limit is exhausted. |
| `x-ratelimit-limit-requests` | `600` | Maximum requests permitted before the RPM limit is exhausted. |
| `x-ratelimit-remaining-requests` | `59` | Remaining requests permitted before the RPM limit is exhausted. |
## Considerations
Rate limits are designed to protect the shared endpoint from overload, not to limit consumption. Because of this, there might be instances where actual throughput exceeds your configured TPM limits.
## Next steps
- [Get started with Serverless Inference](/quickstart/getting-started-with-serverless-inference)
- [Browse available models](/serverless-inference/available-models)
- [Monitor inference metrics](/serverless-inference/inference-metrics)
- [Compare with Self-Serve Deployments](/self-serve-deployments/overview)
---
# Available models
You can use the OpenAI-API compatible endpoint at `api.inference.crusoecloud.com` to access the models below for [Serverless Inference](/quickstart/getting-started-with-serverless-inference). You can also interact with all of the models using the Intelligence Foundry's [chat interface](https://console.crusoecloud.com/foundry/chat/new). All Meta models provided by Crusoe are "Built with Llama".
For each model's pricing information, see [pricing](https://www.crusoe.ai/cloud/pricing#Serverless-Inference).
| MODEL | PROVIDER | TYPE | CONTEXT LENGTH | LICENSE | ACCEPTABLE USE POLICY |
| ------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| [deepseek-ai/DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324) | DeepSeek | instruct | 160k | [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/main/LICENSE) | |
| [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) | DeepSeek | instruct | 1M | [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/LICENSE) | |
| [deepseek-ai/DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) | DeepSeek | instruct | 1M | [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813/blob/main/LICENSE) | |
| [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31B-it) | Google | instruct | 262k | [Apache License 2.0](https://ai.google.dev/gemma/apache_2) | |
| [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) | Meta | instruct | 128k | [Llama 3.3 Community License Agreement](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/LICENSE) | [Llama 3.3 Acceptable Use Policy](https://www.llama.com/llama3_3/use-policy/) |
| [moonshotai/Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) | Moonshot | instruct | 256K | [Modified MIT License](https://huggingface.co/moonshotai/Kimi-K2.6/blob/main/LICENSE) | |
| [nvidia/Nemotron-3-Nano-30B-A3B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8) | NVIDIA | instruct | 262k | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [nvidia/Nemotron-3-Nano-Omni-Reasoning-30B-A3B](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8) | NVIDIA | instruct | 262k | [NVIDIA Open Model Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/) | |
| [nvidia/Nemotron-3-Super-120B-A12B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) | NVIDIA | instruct | 262k | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [nvidia/Nemotron-3-Ultra-550B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4) | NVIDIA | instruct | 262k | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [nvidia/Nemotron-3-VoiceChat](https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B) | NVIDIA | speech-to-speech | 131k | [NVIDIA Software and Model Evaluation License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-and-model-evaluation-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-and-model-evaluation-license/) |
| [nvidia/nemotron-3.5-lightning-30b-a3b](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) | NVIDIA | instruct | 1M | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) | OpenAI | instruct | 128k | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) | [Acceptable Use Policy](https://huggingface.co/openai/gpt-oss-120b/blob/main/USAGE_POLICY) |
| [qwen/Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) | Qwen | instruct | 131k | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE) |
| [qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | Qwen | instruct | 256k | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE) |
| [zai/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1) | Z.ai | instruct | 202k | [MIT License](https://huggingface.co/zai-org/GLM-5.1/blob/main/LICENSE) | |
| [zai/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) | Z.ai | instruct | 256k | [MIT License](https://huggingface.co/zai-org/GLM-5.2/blob/main/LICENSE) | |
| [zai/GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) | Z.ai | instruct | 1M | [MIT License](https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE) | |
| [zai/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) | Z.ai | instruct | 1M | [MIT License](https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/LICENSE) | |
---
# Metrics
Serverless Inference records metrics for every model it serves, with no configuration required. The metrics update every minute and are available on the [metrics](https://console.crusoecloud.com/foundry/metrics) page. You can also integrate them with Grafana dashboards through a Prometheus-compatible query API. To query that API yourself, see [Retrieve metrics using the PromQL API](#retrieve-metrics-using-the-promql-api).
## Available metrics
The following queries return the same series that back the charts on the console metrics page. Replace `{project_id}` with your project ID and `{model_alias}` with the model name you pass in your API requests.
### Requests
`inference_counter_chat_request` counts only successfully served requests. Use `inference_counter_all_chat_request` to include error and rate-limited responses.
| **Metric** | **Definition** | **Metric query** |
| ----------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Request rate | Successful requests per second | `sum by (model_alias) ( rate( inference_counter_chat_request{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )` |
| Requests by status code | All requests, including client errors, rate limits (429), and server errors, split by HTTP status | `sum by (model_alias, status_code) ( rate( inference_counter_all_chat_request{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )` |
| Total requests | Cumulative count of all requests over the query window | `sum by (model_alias) ( increase( inference_counter_all_chat_request{project_id="{project_id}", model_alias="{model_alias}"}[24h] ) )` |
### Tokens
These queries cover input and output token throughput, prefix cache effectiveness, and cumulative totals over a window.
| **Metric** | **Definition** | **Metric query** |
| ----------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Input token rate | Input (prompt) tokens processed per second | `sum by (model_alias) ( rate( inference_counter_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )` |
| Output token rate | Output (completion) tokens generated per second | `sum by (model_alias) ( rate( inference_counter_output_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )` |
| Cached input tokens | Input tokens served from the prefix cache, per second | `sum by (model_alias) ( rate( inference_counter_cached_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )` |
| Prefix cache hit rate | Cached input tokens as a percentage of all input tokens | `( sum by (model_alias) ( rate( inference_counter_cached_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) / sum by (model_alias) ( rate( inference_counter_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 100` |
| Tokens per minute (TPM) | Total tokens (input plus output) processed per minute | `( sum by (model_alias) ( rate( inference_counter_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) + sum by (model_alias) ( rate( inference_counter_output_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 60` |
| Total input tokens | Cumulative input tokens over the query window | `sum by (model_alias) ( increase( inference_counter_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[24h] ) )` |
| Total output tokens | Cumulative output tokens over the query window | `sum by (model_alias) ( increase( inference_counter_output_token{project_id="{project_id}", model_alias="{model_alias}"}[24h] ) )` |
### Latency
Latency histograms record values in seconds. Multiply by 1000 for milliseconds, as the console does. Each query below returns the median—substitute `0.9`, `0.95`, or `0.99` for `0.5` to get other percentiles.
Every histogram on this page, including the per-request token histograms below, also exposes `_sum` and `_count` series, so an average over a window is `sum(increase(_sum[1h])) / sum(increase(_count[1h]))`.
| **Metric** | **Definition** | **Metric query** |
| ---------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Time to first token (TTFT) | Time from request receipt to the first returned token | `histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_first_token_latency_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 1000` |
| Time per output token (TPOT) | Time between successive output tokens | `histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_output_token_latency_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 1000` |
| End-to-end latency | Total request duration, from request receipt to completed response | `histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_chat_latency_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 1000` |
### Tokens per request
These histograms show how large individual requests are. As with the latency histograms, substitute another quantile for `0.5` to see a different percentile.
| **Metric** | **Definition** | **Metric query** |
| ------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Input tokens per request | Distribution of input (prompt) tokens per request | `histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_input_tokens_per_request_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) )` |
| Output tokens per request | Distribution of output (completion) tokens per request | `histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_output_tokens_per_request_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) )` |
## Labels
Use these labels to filter and group the queries above:
| **Label** | **Description** |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `project_id` | The project billed for the request |
| `model_alias` | The model name you pass in your API request—filter on this label |
| `model_name` | The deployment serving the request: the upstream model name plus a deployment suffix, such as `google-gemma-4-31b-it-0dc46fd7`. It doesn't match `model_alias` |
| `service_tier` | The service tier that handled the request |
| `status_code` | HTTP status code of the response. Present on both request counters, but `inference_counter_chat_request` only ever reports `200`, so query `inference_counter_all_chat_request` for errors |
| `is_streaming` | Whether the request used streaming |
## Retrieve metrics using the PromQL API
Query the metrics API endpoint directly to retrieve data for a single instant or a specific time range. The API endpoint is:
```text
https://api.cloud.crusoe.ai/v1/projects//metrics/timeseries
```
To find your project ID, navigate to the [projects](https://console.crusoecloud.com/projects) page in the console, and copy the project ID.
### Generate a monitoring token
Querying metrics requires a monitoring token. To generate one, run the following command with the [Crusoe CLI](/installing-the-cli):
```sh
crusoe monitoring tokens create
```
This command generates an `API-Key` that authenticates your requests to the metrics API. Store the token in a secret or key management tool, because you can't retrieve it later.
### Query metrics
To retrieve the most recent TTFT data point in your project, run:
```sh
curl -G https://api.cloud.crusoe.ai/v1/projects//metrics/timeseries \
--data-urlencode 'query=histogram_quantile(0.5, sum by (model_alias, le) (rate(inference_histogram_first_token_latency_bucket[5m]))) * 1000' \
-H 'Authorization: Bearer '
```
### Import data into Grafana
To chart Serverless Inference metrics alongside the rest of your telemetry, add a Prometheus data source to your own Grafana instance. You need the `API-Key` from [Generate a monitoring token](#generate-a-monitoring-token).
1. Set **Prometheus server URL** to `https://api.cloud.crusoe.ai/v1/projects//metrics/timeseries`.
2. Under **Authentication → HTTP headers**, add the following header:
```text
Header: Authorization
Value: Bearer
```
---
# Self-Serve Deployments
[Self-serve deployments](https://console.crusoecloud.com/foundry/deployments) provide reserved inference capacity on Crusoe's optimized inference engine and managed infrastructure. They give you predictable performance, dedicated throughput, and scalability that grows with your workload.
## When to use self-serve deployments
Self-serve deployments are best when you need:
- **Predictability** — Consistent, predictable inference performance over time
- **Sustained traffic** — High-volume processing that benefits from reduced cost-per-token as you scale
- **Fine-tuned models** — Deployment of your own Low-Rank Adaptation (LoRA) adapters trained through [Serverless Fine-Tuning](/serverless-fine-tuning/overview) offering
- **Control over scaling** — Reduced risk of rate limiting and direct control over the number of replicas supporting your workload
- **Pay-per-use flexibility** — GPU-hour billing that lets you align cost with actual usage
### Self-serve vs. serverless
If your workload is sporadic or low-volume, serverless might be a better fit. Use the following table to compare the two options across the dimensions that most affect cost and performance.
| Dimension | Self-serve | Serverless |
| ------------------- | --------------------------------------------------------- | ------------------------------------------------------- |
| **Capacity** | Dedicated deployment with reserved GPUs | Shared multi-tenant pool |
| **Rate limits** | None; throughput bounded by replica count | [Shared pool limits](/serverless-inference/rate-limits) |
| **Billing** | GPU-hour (time-based) | Per-token |
| **Model support** | Base models and LoRA adapters from Serverless Fine-Tuning | Base models only; no custom or fine-tuned models |
| **Cost efficiency** | Best at sustained high utilization | Best for sporadic or low-volume use |
## Choose an optimization profile
Every self-serve deployment is configured with one optimization profile per model. The profile determines how the inference engine is tuned for your workload.
| Profile | Optimization | Best for |
| ------------------ | ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| **Responsiveness** | Low latency, optimized for quick responses | Interactive applications, real-time inference, latency-sensitive workloads |
| **Throughput** | Cost efficiency at scale, optimized for token volume | Batch processing, high-volume workflows, cost-per-token minimization |
| **Balanced** | Hybrid blend of throughput and responsiveness, optimized to support moderate token volume and latency | General purpose production traffic |
## Supported models
Refer to [Available models](/self-serve-deployments/available-models) for the full list of supported base models, which you can also deploy with LoRA adapters trained through [Serverless Fine-Tuning](/serverless-fine-tuning/overview).
## Next steps
- [Set up your first deployment](/self-serve-deployments/quickstart)
- For additional optimization, [Contact us](https://www.crusoe.ai/contact-sales) about Tailored Deployments
- Learn more about [Managed AI](/managed-ai/overview)
---
# Deploy a self-serve deployment
Self-serve deployments give you reserved inference capacity on Crusoe's optimized inference engine and managed infrastructure. You choose a base model (or a fine-tuned adapter) and a deployment configuration, and Crusoe handles engine selection, tuning, autoscaling, and rate limiting. You get predictable performance, dedicated throughput, no shared rate limits, and per-GPU-hour billing you control.
## Prerequisites
- Install the OpenAI Python client (`pip install openai httpx`), if you use the Python examples.
- For Low-Rank Adaptation (LoRA) adapters, bring a checkpoint from a successful training job completed with [Serverless Fine-Tuning](/serverless-fine-tuning/overview).
## 1. Log in or create an account
Log in to the [Crusoe Cloud Console](https://console.crusoecloud.com) or [Create an account](/create-an-account).
After you log in, switch to the **Intelligence Foundry** app in the bottom-left of the [console](https://console.crusoecloud.com/).
## 2. Generate an API key and authenticate
To create an API key through the [console](https://console.crusoecloud.com):
1. From the [console](https://console.crusoecloud.com), click **Admin** in the bottom-left corner.
2. Select **Security** > **[Intelligence API keys](https://console.crusoecloud.com/security/inference-api-keys)** from the left navigation.
3. Click **Create**.
4. (Optional) Enter an alias for your key.
5. (Optional) Enter an expiration date for your key.
6. Copy the **API key**. Make sure that you save the key in a secure location before leaving the page.
### Authenticate against the API
Use your Intelligence API key to authenticate across your intelligence API calls and self-serve deployment management API calls.
1. Export the token and base URL in your shell:
```shell
export API_TOKEN=''
export INFERENCE_URL='https://api.inference.crusoecloud.com/v1/chat/completions'
export DEPLOYMENT_URL='https://api.crusoecloud.com/v1/projects/{project_id}/foundry/selfserve/'
```
2. For inference requests, construct an OpenAI client pointed at the Crusoe gateway. Every Python example on this page assumes you have this `openai_client` in scope:
```python
from openai import OpenAI
import httpx, os
openai_client = OpenAI(
api_key=os.environ["API_TOKEN"],
url=f"os.environ['INFERENCE_URL']",
http_client=httpx.Client(proxy=None, trust_env=False),
)
```
The full OpenAPI specification is published at [api.intelligence.crusoecloud.com/docs](https://api.intelligence.crusoecloud.com/docs).
## 3. Choose a base model and deployment configuration
Define your deployment by selecting the base model you want to serve and the deployment configuration you want to optimize for.
### Deployment configurations
Each deployment offers one or more optimization profiles. Pick the configuration that matches your workload requirements and Crusoe will apply the corresponding engine, hardware, and optimizations for you. There's no hand-tuning required.
| Configuration | Optimization | Best for |
| ------------------ | ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| **Responsiveness** | Low latency, optimized for time-to-first-token | Interactive applications, real-time inference, latency-sensitive workloads |
| **Throughput** | Cost efficiency at scale, optimized for token volume | Batch processing, high-volume workflows, cost-per-token minimization |
| **Balanced** | Hybrid blend of throughput and responsiveness, optimized to support moderate token volume and latency | General purpose production traffic |
### Supported models
Refer to [Available models](/self-serve-deployments/available-models) for the full list of supported base models, which you can also deploy with LoRA adapters trained through [Serverless Fine-Tuning](/serverless-fine-tuning/overview).
## 4. Create a deployment
Create a deployment from the console or API to get started.
**UI:**
1. Sign in to the [console](https://console.crusoecloud.com/) and switch to the **Intelligence Foundry** app in the bottom-left corner.
2. Select **[Self-Serve Deployments](https://console.crusoecloud.com/foundry/deployments)** from the **Inference** section of the left navigation. The page lists every deployment in your project with its status, model, hardware, replicas, and metadata.
3. Click **Create deployment** and complete the form:
- Select a base model and, optionally, a fine-tuned checkpoint
- Select a deployment configuration (Responsiveness, Throughput, or Balanced)
- Select the number of replicas you want your deployment to support
The hourly cost associated with your chosen deployment configuration is displayed before you confirm.
**cURL:**
```bash
curl "$DEPLOYMENT_URL/deployments" \
-X POST \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"flavor_id": "",
"deployment_name": "",
"replicas":
}'
```
After the deployment creation is initiated, your new deployment will appear in the self-serve deployment table. Click any row in the self-serve deployment table to view activity logs, endpoint metadata, and its current replica count.
## 5. Check deployment status
A new deployment provisions reserved capacity, which can take up to 40 minutes to complete. A deployment moves through the following states during its lifecycle:
| State | Description |
| -------------- | ------------------------------------------------------------ |
| `Creating` | Capacity is being provisioned and the engine is starting up. |
| `Ready` | The deployment is ready to serve traffic. |
| `Scaling up` | The deployment is scaling up its active replicas. |
| `Scaling down` | The deployment is scaling down its active replicas. |
| `Syncing` | The deployment alias is being updated. |
| `Failed` | The deployment couldn't be created or updated. |
| `Deleting` | The deployment is being torn down. |
| `Deleted` | The deployment has been removed. |
Check the status of your deployment from the deployment table on the **Self-Serve Deployments** page or directly through the API. The status updates to `Ready` when the deployment is available to serve traffic.
```bash
curl "$DEPLOYMENT_URL/deployments" \
-X GET \
-H "Authorization: Bearer $API_TOKEN" \
```
## 6. Run inference
When the deployment is in `Ready` state, send requests to it through the OpenAI-compatible Chat Completions API using your API key. Pass the deployment alias as the `model`.
**python:**
```python
import os
from openai import OpenAI
client = OpenAI(
base_url=os.environ['INFERENCE_URL'],
api_key=os.environ['API_TOKEN'],
)
response = client.chat.completions.create(
model='',
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Summarize the theory of relativity in one sentence."}
],
)
print(response.to_json())
```
**typescript:**
```typescript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: process.env.INFERENCE_URL,
apiKey: process.env.API_TOKEN,
});
client.chat.completions
.create({
model: "",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{
role: "user",
content: "Summarize the theory of relativity in one sentence.",
},
],
})
.then((response) => console.log(response));
```
**cURL:**
```shell
curl $INFERENCE_URL \
--request 'POST' \
--header 'Content-Type: application/json' \
--header 'Accept: text/event-stream' \
--header "Authorization: Bearer $API_TOKEN" \
--data '{
"model": "",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Summarize the theory of relativity in one sentence."}
]
}'
```
Because your deployment runs on reserved capacity, its throughput is bounded by replica count rather than a shared rate limit. To add headroom for traffic spikes, increase the replica count on the deployment in the next step.
## 7. Manage deployments
You can update a running deployment (for example, to change replica count or the deployment alias) or delete one you no longer need. When you delete a deployment, billing stops.
To view all deployment management options, select the three-dot icon on any deployment row on the **Self-Serve Deployments** page. The menu exposes options to edit the deployment alias, update the replica count, or delete the deployment.
### Configure notifications
By default, [notifications](/managed-ai/notifications) are sent when a self-serve deployment is created, deleted, or scaled. You can manage your preferences from the console's [Notifications settings](https://console.crusoecloud.com/notifications) page.
### Edit a deployment alias
To update the alias for a deployment:
**UI:**
1. Define a unique name for your deployment.
2. Confirm your new deployment name.
**cURL:**
```bash
curl "$DEPLOYMENT_URL/deployments/{id}" \
-X PATCH \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"deployment_name": ""
}'
```
The deployment status updates to `Syncing` while the alias is updated, and returns to `Ready` when the update is complete.
### Update replica counts
To adjust the number of replicas for a deployment:
**UI:**
1. Select a value that reflects your expected traffic load and fits within your allotted quota.
2. Confirm your new replica count.
**cURL:**
```bash
curl "$DEPLOYMENT_URL/deployments/{id}" \
-X PATCH \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"replicas": ""
}'
```
The deployment status updates to `Scaling up` or `Scaling down` based on the direction of the change, and returns to `Ready` when the update is complete. The deployment can still serve traffic while the replica count is being adjusted.
### Delete a deployment
After you confirm deletion, the deployment status updates to `Deleting`, and the deployment disappears from the list once the deletion is complete. Billing stops when the deployment is deleted.
```bash
curl "$DEPLOYMENT_URL/deployments/{id}" \
-X DELETE \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
```
### View deployment details
To view the deployment overview, activity log, metadata, and sample code for inferencing, select the deployment alias from the **Self-Serve Deployments** page or use the API.
```bash
curl "$DEPLOYMENT_URL/deployments/{id}" \
-X GET \
-H "Authorization: Bearer $API_TOKEN"
```
## 8. (Optional) Deploy a fine-tuned model
Self-serve works with Crusoe serverless fine-tuning to help you get your tuned models to production with one click. A LoRA adapter you train there is registered in the same model registry as the base models, so it appears in the models list as soon as training completes.
You have three ways to deploy a fine-tuned checkpoint:
- **From the Self-Serve Deployments page:** Follow the [Create a deployment](#4-create-a-deployment) steps, then select your fine-tuned checkpoint from the list after you select the corresponding base model architecture.
- **From a Fine-tuned model's [Jobs](https://console.crusoecloud.com/foundry/fine-tuning/jobs) page:** Click the three-dot menu next to the checkpoint you want to deploy and select **Deploy**.
- **From the self-serve deployments API:** First, retrieve your `fine_tuned_model` identifier for your desired fine-tuned model checkpoint and include that in your self-serve deployment creation request.
```bash
curl "$DEPLOYMENT_URL/deployments" \
-X POST \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"fine_tuned_model_id": "",
"flavor_id": "",
"deployment_name": "",
"replicas":
}'
```
Because fine-tuning and deployment share the same registry and API conventions, you can iterate quickly: train a new adapter, point a new deployment at it, and shift traffic—without rebuilding your serving stack.
## Next steps
- Need optimization beyond the standard configurations? [Contact us](https://www.crusoe.ai/contact-sales) about Tailored Deployments
- For an overview of Crusoe's Managed AI options, see [Managed AI](/managed-ai/overview)
---
# Available models
Refer to the table on this page to see which base models [Self-Serve Deployments](/self-serve-deployments/quickstart) supports. You can also deploy these models with LoRA adapters trained through [Serverless Fine-Tuning](/serverless-fine-tuning/overview).
For each model's pricing information, see [pricing](https://www.crusoe.ai/cloud/pricing#Self-Serve-Deployments).
| MODEL | PROVIDER | TYPE | CONTEXT LENGTH | LICENSE | ACCEPTABLE USE POLICY |
| ------------------------------------------------------------------------------------------------------------------ | -------- | -------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) | DeepSeek | instruct | 1M | [MIT License](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/LICENSE) | |
| [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31B-it) | Google | instruct | 262k | [Apache License 2.0](https://ai.google.dev/gemma/apache_2) | |
| [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | Meta | instruct | — | [Llama 3.1 Community License Agreement](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE) | [Llama 3.1 Acceptable Use Policy](https://www.llama.com/llama3_1/use-policy/) |
| [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) | Meta | instruct | 131k | [Llama 3.3 Community License Agreement](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/LICENSE) | [Llama 3.3 Acceptable Use Policy](https://www.llama.com/llama3_3/use-policy/) |
| [nvidia/nemotron-3.5-lightning-30b-a3b](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) | NVIDIA | instruct | 1M | [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) | [NVIDIA Acceptable Use Terms](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) |
| [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | OpenAI | instruct | — | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) | [Acceptable Use Policy](https://huggingface.co/openai/gpt-oss-20b/blob/main/USAGE_POLICY) |
| [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) | OpenAI | instruct | 131k | [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) | [Acceptable Use Policy](https://huggingface.co/openai/gpt-oss-120b/blob/main/USAGE_POLICY) |
| [qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-8B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-8B/blob/main/LICENSE) |
| [qwen/Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) | Qwen | instruct | 262k | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE) |
| [qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/LICENSE) |
| [qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE) |
| [qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE) |
| [qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | Qwen | instruct | — | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE) |
| [qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | Qwen | instruct | 256k | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE) | [Apache License 2.0](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE) |
| [zai/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) | Z.ai | instruct | 1M | [MIT License](https://huggingface.co/zai-org/GLM-5.2/blob/main/LICENSE) | |
---
# Notifications
Our [notifications](https://console.crusoecloud.com/notifications) service sends alerts for [Managed AI](/managed-ai/overview) events that affect your resources and deployments. These [notification events](#notification-event-categories) include [budget alerts](/usage-billing/budget-alerts) for exceeded budget thresholds and [Self-serve deployment](/self-serve-deployments/overview) status updates. You can manage your notification preferences using the console, and [configure webhooks](#route-notifications-to-slack-and-webhooks) for Slack—and other webhook endpoints—on the Svix dashboard, which you can access from the console.
:::note
This page is specific to Managed AI events, if you need to configure notifications for Infrastructure Cloud events for an organization, see [Notifications](/notifications/overview).
:::
## How the notification service works
Crusoe Cloud continuously monitors your usage and deployments. When a significant event is detected, the event is published to a notification pipeline and routed to your configured channels with context.
### Notification event categories
Crusoe Cloud groups notification events into the following categories for Managed AI in the console.
:::important
The following categories are specific to Managed AI events. To view information for additional categories that apply to Infrastructure Cloud events, see [Notifications](/notifications/overview).
:::
| Category | Description |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Payment Method & Budget Alerts** | Notifications that help you monitor and control your cloud spending. See [Budget Alerts](/usage-billing/budget-alerts) for details. |
| **Managed Intelligence** | Notifications for [Self-serve deployment](/self-serve-deployments/quickstart) status updates, including: `deployment created`, `deployment scaled`, and `deployment deleted`. |
## Access console and email notifications
Console notifications are enabled by default and don't require configuration.
### View notifications in the Crusoe Cloud Console
To view notifications in the [console](https://console.crusoecloud.com/):
1. From the [console](https://console.crusoecloud.com/), click the bell icon in the top-right corner to view the latest, unread notifications. You can also dismiss notifications from this view.
2. Select **[All Notifications](https://console.crusoecloud.com/notifications)** from the modal to view your complete notification history. Previously dismissed notifications still appear in this view.
### Receive email notifications
When email notifications are enabled for an event, notifications are sent automatically when the event occurs.
## Route notifications to Slack and webhooks
To deliver notifications to Slack or an external webhook endpoint (for example, PagerDuty, Opsgenie, or custom automation), configure a delivery endpoint in the Svix dashboard (through the Crusoe Cloud Console).
### Configure Slack notifications
Route notifications to a Slack channel so your on-call team sees alerts in real time.
1. From the [console](https://console.crusoecloud.com/), click the bell icon in the top-right corner.
2. Click **[All Notifications](https://console.crusoecloud.com/notifications)**, then click **Manage Slack/Webhook** in the top-right corner. This links out to a separate page.
3. On the new page, click **Add Endpoint**.
4. Select **Slack** as the endpoint type.
5. Provide your Slack incoming webhook URL or click **Connect to Slack** (requires authentication). To generate a webhook URL, follow the [Slack documentation on incoming webhooks](https://api.slack.com/messaging/webhooks).
:::note
If the authorization to **Connect to Slack** or the incoming webhook URL requires approval from your enterprise Slack account, reach out to your IT department to authorize the Svix app for your account.
:::
6. Select which event types to subscribe to.
7. Click **Create**.
### Configure webhook notifications
To integrate with PagerDuty, Opsgenie, custom automation, or other tools, configure a generic webhook endpoint.
1. From the [console](https://console.crusoecloud.com/), click the bell icon in the top-right corner.
2. Click **[All Notifications](https://console.crusoecloud.com/notifications)**, then click **Manage Slack/Webhook** in the top-right corner. This links out to a separate page.
3. Click **Add Endpoint**.
4. Select **Webhook** as the endpoint type.
5. Provide your webhook endpoint URL.
6. Select which event types to subscribe to.
7. Click **Create**.
Webhook payloads are delivered as HTTP POST requests with a JSON body containing the event details. The exact structure varies by event type.
## What's next
- [Managed AI](/managed-ai/overview)—Learn more about Managed AI services
- [Self-serve deployments](/self-serve-deployments/overview)—Learn how to self-serve a deployment for inference
- [Serverless fine-tuning](/serverless-fine-tuning/overview)—Fine-tune a model before serving it with self-serve deployments
---
# Infrastructure Cloud
Crusoe's Infrastructure Cloud gives you the building blocks—[Compute](/compute), [Networking](/networking), [Storage](/storage), and [Orchestration](/orchestration)—to run training and inference workloads on infrastructure you manage.
- [Compute](/compute): Provision GPU and CPU virtual machines, instance templates, and images.
- [Storage](/storage): Attach persistent disks and manage object storage buckets for your workloads.
- [Networking](/networking): Configure VPC networks, subnets, firewall rules, and load balancers.
- [Orchestration](/orchestration): Run distributed workloads on managed Kubernetes (CMK) or Slurm clusters.
- [Container Registry](/container-registry): Store and serve container images from a Crusoe-hosted registry.
---
# Overview
Crusoe Cloud provides a high-performance, sustainable cloud platform with the latest hardware.
## Virtual Machine (VM) specifications
### GPU-enabled VMs
Customers can create GPU-enabled VMs with the following specs:
| Type | vCPU | GPU | Memory | Ephemeral Disk | VPC Network | InfiniBand Network | Block Storage Network | Zones |
| ---------------------- | -------------------------------------- | ------------------------- | ------ | -------------- | ----------- | ------------------ | --------------------- | --------------------------------------------- |
| `mi355x-288gb-roce.8x` | 240 vCPUs AMD EPYC (Turin) | 8x AMD MI355x 288GB OAM | 3000GB | 8x 3.84TB NVMe | 175 Gbps | 3200 Gbps | 25 Gbps | us-east2-a |
| `b300-288gb-sxm-ib.8x` | 240 vCPUs Intel Xeon (Granite Rapids) | 8x Nvidia B300 288GB SXM | 3100GB | 8x 3.84TB NVMe | 175 Gbps | 6400 Gbps | 25 Gbps | eu-iceland1-a |
| `gb200-186gb-nvl-4x` | 128 vCPUs Nvidia ARM Neoverse (Grace) | 4x Nvidia GB200 186GB NVL | 886GB | 4x 1.92TB NVMe | 175 Gbps | 1600 Gbps | 25 Gbps | eu-iceland1-a |
| `b200-180gb-sxm-ib-8x` | 176 vCPUs Intel Xeon (Emerald Rapids) | 8x Nvidia B200 180GB SXM | 3000GB | 8x 1.92TB NVMe | 175 Gbps | 3200 Gbps | 25 Gbps | eu-iceland1-a, eu-norway1-a, us-west1-a |
| `mi300x-192gb-ib.8x` | 240 vCPUs AMD EPYC (Genoa) | 8x AMD MI300X 192GB OAM | 2000GB | 8x 1.92TB NVMe | 175 Gbps | 3200 Gbps | 25 Gbps | us-east1-a |
| `l40s-48gb.10x` | 80 vCPUs AMD EPYC (Genoa) | 10x NVIDIA L40S 48GB PCIe | 1470GB | N/A | 175 Gbps | N/A | 25 Gbps | us-east1-a, us-southcentral1-a |
| `l40s-48gb.8x` | 64 vCPUs AMD EPYC (Genoa) | 8x NVIDIA L40S 48GB PCIe | 1176GB | N/A | 140 Gbps | N/A | 20 Gbps | us-east1-a, us-southcentral1-a |
| `l40s-48gb.4x` | 32 vCPUs AMD EPYC (Genoa) | 4x NVIDIA L40S 48GB PCIe | 588GB | N/A | 70 Gbps | N/A | 10 Gbps | us-east1-a, us-southcentral1-a |
| `l40s-48gb.2x` | 16 vCPUs AMD EPYC (Genoa) | 2x NVIDIA L40S 48GB PCIe | 294GB | N/A | 35 Gbps | N/A | 5 Gbps | us-east1-a, us-southcentral1-a |
| `l40s-48gb.1x` | 8 vCPUs AMD EPYC (Genoa) | 1x NVIDIA L40S 48GB PCIe | 147GB | N/A | 17.5 Gbps | N/A | 2.5 Gbps | us-east1-a, us-southcentral1-a |
| `h200-141gb-sxm-ib.8x` | 176 vCPUs Intel Xeon (Sapphire Rapids) | 8x NVIDIA H200 141GB SXM5 | 2000GB | 8x 1.92TB NVMe | 175 Gbps | 3200 Gbps | 25 Gbps | eu-iceland1-a |
| `h100-80gb-sxm-ib.8x` | 176 vCPUs Intel Xeon (Sapphire Rapids) | 8x NVIDIA H100 80GB SXM5 | 960GB | 8x 960GB NVMe | 175 Gbps | 3200 Gbps | 25 Gbps | us-east1-a, us-southcentral1-a, eu-iceland1-a |
| `a100-80gb-sxm-ib.8x` | 96 vCPUs Intel Xeon (Ice Lake) | 8x NVIDIA A100 80GB SXM4 | 960GB | 8x 960GB NVMe | 175 Gbps | 1600 Gbps | 25 Gbps | us-east1-a |
| `a100-80gb.8x` | 96 vCPUs Intel Xeon (Ice Lake) | 8x NVIDIA A100 80GB PCIe | 960GB | 8x 960GB NVMe | 175 Gbps | N/A | 25 Gbps | us-east1-a |
| `a100-80gb.4x` | 48 vCPUs Intel Xeon (Ice Lake) | 4x NVIDIA A100 80GB PCIe | 480GB | 4x 960GB NVMe | 87.5 Gbps | N/A | 12.5 Gbps | us-east1-a |
| `a100-80gb.2x` | 24 vCPUs Intel Xeon (Ice Lake) | 2x NVIDIA A100 80GB PCIe | 240GB | 2x 960GB NVMe | 43.75 Gbps | N/A | 6.25 Gbps | us-east1-a |
| `a100-80gb.1x` | 12 vCPUs Intel Xeon (Ice Lake) | 1x NVIDIA A100 80GB PCIe | 120GB | 1x 960GB NVMe | 21.875 Gbps | N/A | 3.125 Gbps | us-east1-a |
### CPU VMs
Crusoe Cloud offers CPU-only instance types in two families: `c1a`/`c2a` ("general purpose", intended for control planes, web serving, and other lightweight compute) and `s1a`/`s2a` ("storage optimized", intended for running high performance file systems like [Lustre](https://www.lustre.org) or [ceph](https://ceph.io), or object stores like [minio](https://min.io/)). `c1a` and `s1a` are available in all regions except `eu-norway1-a`, which uses `c2a` and `s2a`. New regions will use `c2a` and `s2a` going forward.
Customers can create `c1a` VMs with the following specs:
| Type | vCPU | Memory | Disk | VPC Network | Block Storage Network | Zones |
| ---------- | --------------------- | ------ | ---- | ----------- | --------------------- | ----------------------------------------------------------------------------- |
| `c1a.2x` | 2 vCPUs AMD (Genoa) | 8GB | N/A | 1 Gbps | 512 Mbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `c1a.4x` | 4 vCPUs AMD (Genoa) | 16GB | N/A | 2 Gbps | 1 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `c1a.8x` | 8 vCPUs AMD (Genoa) | 32GB | N/A | 5 Gbps | 2 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `c1a.16x` | 16 vCPUs AMD (Genoa) | 64GB | N/A | 10 Gbps | 4 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `c1a.32x` | 32 vCPUs AMD (Genoa) | 128GB | N/A | 20 Gbps | 10 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `c1a.64x` | 64 vCPUs AMD (Genoa) | 256GB | N/A | 35 Gbps | 20 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `c1a.128x` | 128 vCPUs AMD (Genoa) | 512GB | N/A | 70 Gbps | 40 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `c1a.176x` | 176 vCPUs AMD (Genoa) | 704GB | N/A | 100 Gbps | 50 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
Customers can create `s1a` VMs with the following specs:
| Type | vCPU | Memory | Ephemeral Disk | VPC Network | Block Storage Network | Zones |
| ---------- | --------------------- | ------ | -------------- | ----------- | --------------------- | ----------------------------------------------------------------------------- |
| `s1a.20x` | 20 vCPUs AMD (Genoa) | 176GB | 1x 12.8TB NVMe | 25 Gbps | up to 12.5 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `s1a.40x` | 40 vCPUs AMD (Genoa) | 352GB | 2x 12.8TB NVMe | 50 Gbps | up to 25 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `s1a.60x` | 60 vCPUs AMD (Genoa) | 528GB | 3x 12.8TB NVMe | 75 Gbps | up to 37.5 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `s1a.80x` | 80 vCPUs AMD (Genoa) | 704GB | 4x 12.8TB NVMe | 100 Gbps | up to 50 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `s1a.120x` | 120 vCPUs AMD (Genoa) | 1056GB | 6x 12.8TB NVMe | 150 Gbps | up to 75 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
| `s1a.160x` | 160 vCPUs AMD (Genoa) | 1408GB | 8x 12.8TB NVMe | 200 Gbps | up to 100 Gbps | us-east1-a, us-southcentral1-a, us-northcentral1-a, eu-iceland1-a, us-west1-a |
Customers can create `c2a` VMs with the following specs:
| Type | vCPU | Memory | Disk | VPC Network | Block Storage Network | Zones |
| ---------- | --------------------- | ------ | ---- | ----------- | --------------------- | ------------ |
| `c2a.2x` | 2 vCPUs AMD (Turin) | 8GB | N/A | 2 Gbps | 2 Gbps | eu-norway1-a |
| `c2a.4x` | 4 vCPUs AMD (Turin) | 16GB | N/A | 4 Gbps | 4 Gbps | eu-norway1-a |
| `c2a.8x` | 8 vCPUs AMD (Turin) | 32GB | N/A | 9 Gbps | 9 Gbps | eu-norway1-a |
| `c2a.16x` | 16 vCPUs AMD (Turin) | 64GB | N/A | 18 Gbps | 18 Gbps | eu-norway1-a |
| `c2a.32x` | 32 vCPUs AMD (Turin) | 128GB | N/A | 37 Gbps | 37 Gbps | eu-norway1-a |
| `c2a.64x` | 64 vCPUs AMD (Turin) | 256GB | N/A | 74 Gbps | 74 Gbps | eu-norway1-a |
| `c2a.128x` | 128 vCPUs AMD (Turin) | 512GB | N/A | 148 Gbps | 148 Gbps | eu-norway1-a |
| `c2a.172x` | 172 vCPUs AMD (Turin) | 688GB | N/A | 200 Gbps | 200 Gbps | eu-norway1-a |
Customers can create `s2a` VMs with the following specs:
| Type | vCPU | Memory | Ephemeral Disk | VPC Network | Block Storage Network | Zones |
| --------- | -------------------- | ------ | --------------- | ----------- | --------------------- | ------------ |
| `s2a.10x` | 10 vCPUs AMD (Turin) | 86GB | 1x 15.36TB NVMe | 25 Gbps | 25 Gbps | eu-norway1-a |
| `s2a.20x` | 20 vCPUs AMD (Turin) | 172GB | 2x 15.36TB NVMe | 50 Gbps | 50 Gbps | eu-norway1-a |
| `s2a.40x` | 40 vCPUs AMD (Turin) | 344GB | 4x 15.36TB NVMe | 100 Gbps | 100 Gbps | eu-norway1-a |
| `s2a.60x` | 60 vCPUs AMD (Turin) | 516GB | 6x 15.36TB NVMe | 150 Gbps | 150 Gbps | eu-norway1-a |
| `s2a.80x` | 80 vCPUs AMD (Turin) | 688GB | 8x 15.36TB NVMe | 200 Gbps | 200 Gbps | eu-norway1-a |
:::info
Certain VM types in certain regions are currently restricted and may not be available for immediate provisioning. If you require access, please [contact our sales team](https://crusoe.ai/contact-us#sales) to discuss your use case.
All local storage associated with GPU, `s1a`, and `s2a` instances are considered ephemeral storage, even when configured in a storage cluster. Please see [managing ephemeral disks](https://docs.crusoecloud.com/storage/disks/managing-ephemeral-disks) for more details.
A vCPU represents a single thread on a CPU.
:::
## VM lifecycle
VMs transition through the following states:
| State | Description |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `Creating` | The VM is being created. This can take up to a minute. After the VM has been created, it will transition to the `Starting` state |
| `Starting` | The VM is being started. Once it has started, it will transition to `Running`. |
| `Running` | The VM is running and accessible. You can now SSH into it and run your desired software. The VM will continue to run until you stop it. |
| `Stopping` | The VM is being shut down, and no more customer work will be performed. Once it has shut down, it will transition to `Stopped`. |
| `Stopped` | The VM is stopped and inaccessible. You must manually start it to change the state. Stopped VMs aren't billed for hourly compute charges. |
## VM billing
Crusoe Cloud bills for VM usage in two separate ways:
- **On-demand**
- **Reservations**
On-demand usage provides developers the most flexibility. VMs are charged based on the time the machine is in the `running` state (per second). This includes time that the VM is spent running [Lifecycle Scripts](./managing-lifecycle-scripts.mdx) on startup or shutdown. Stopped VMs are not charged for on-demand usage, but you will still be billed for the 128GB OS disk at $0.08/GiB/month until the VM is deleted.
Reservations offer discounts in exchange for committed usage. These VMs are billed for the entire duration of the commitment, regardless of their state. Learn more about [reservations](../../usage-billing/reservations/overview.md).
A VM may only be billed via one of these mechanisms (on-demand or reservation) at a time, but different VMs in the same organization may be billed differently.
---
# Manage your SSH keys
## Creating an SSH key
If you don't already have an SSH key, you'll need to create one. We recommend following [GitHub's "Generating a new SSH key" documentation](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
## Uploading an SSH key
You will need to upload an SSH key in order to access a VM on Crusoe Cloud. If you are creating a VM for the first time and haven't uploaded an SSH key, you will be prompted to upload one during creation. It will be saved with the name "Default" and accessible on subsequent VM creations. You can add additional keys to be used when creating a VM here.
**UI:**
To upload an SSH key via the [console](https://console.crusoecloud.com):
1. Click on your profile icon on the top right and select **Security**.
2. Select **Security** > **[SSH Keys](https://console.crusoecloud.com/security/ssh-keys)** in the left nav.
3. Click **Add SSH Key**.
4. Add a name for your SSH key.
5. Add the SSH public key (`ssh-rsa...` or `ssh-ed25519...`).
6. Click **Add**.
Adding keys here does not impact who can access a VM, unless that key is used to create the VM.
## Supported SSH key formats
Crusoe Cloud supports all SSH public key formats that are accepted by OpenSSH. These include:
- `sk-ecdsa-sha2-nistp256@openssh.com`
- `ecdsa-sha2-nistp256`
- `ecdsa-sha2-nistp384`
- `ecdsa-sha2-nistp521`
- `sk-ssh-ed25519@openssh.com`
- `ssh-ed25519`
- `ssh-dss`
- `ssh-rsa`
For more information on the authorized key format accepted by OpenSSH, please see their [docs](https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT).
## Deleting an SSH key
If your SSH key has been compromised or is no longer necessary, you should delete it. Deleting an SSH key is a permanent action that cannot be undone.
**UI:**
To delete an SSH key via the [console](https://console.crusoecloud.com):
1. Click on your profile icon on the top right and select **Security**.
2. Select **Security** > **[SSH Keys](https://console.crusoecloud.com/security/ssh-keys)** in the left nav.
3. Navigate to the SSH key you want to delete and click the trash can icon.
4. Click **Delete**.
---
# Manage your VMs
## Creating a new VM
**CLI:**
Use the `compute vms create` command to create a VM of your choice. As an example, you can create a VM that uses a single Nvidia L40S GPU:
```sh
crusoe compute vms create \
--name my-vm \
--type l40s-48gb.1x \
--location us-southcentral1-a \
--image ubuntu22.04:latest \
--keyfile ~/.ssh/id_ed25519.pub
```
You can find possible values for type and location by running `crusoe compute vms types` and `crusoe locations list` respectively.
If you don't specify an image, the VM will default to the latest version of ubuntu 22.04. Run `crusoe compute images list` for more options.
**UI:**
To create a VM via the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Click **Create Instance**.
3. Follow the UI flow to input all required elements.
4. Click **Create**.
**Terraform:**
Creating and accessing VMs is the first step to getting started on Crusoe Cloud. The following is intended to help get you started using Terraform to provision a VM in Crusoe Cloud.
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates a VM that uses a single Nvidia A40 GPU called “my-vm”:
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
// local files
locals {
ssh_key = file("~/.ssh/id_ed25519.pub") # replace with path to your public SSH key if different
}
// new VM
resource "crusoe_compute_instance" "my_vm" {
name = "my-vm"
type = "l40s-48gb.1x"
location = "us-southcentral1-a"
image = "ubuntu22.04:latest" # use the 'latest' flag to get the most up to date image available from Crusoe
ssh_key = local.ssh_key
}
```
`name`, `type`, `location` and `ssh_key` are required arguments. `default_project` is also required, but is typically specified in the config file, which is why it's omitted in the `crusoe_compute_instance` resource (see step 4 above). `image` is not required and for this example, we will use an `ubuntu22.04:latest` image.
After saving the code to a `main.tf` file, the following commands serve as the process to create a resource in Crusoe Cloud using Terraform:
`terraform init` - Initializes a working directory containing Terraform configuration files.
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will create the resources.
You can confirm that terraform successfully created the resources through the console, but if you prefer CLI, you can also run:
`crusoe compute vms list`
Which will show you the VMs you have created in your account.
## Viewing all existing VMs
**CLI:**
Use the `compute vms list` command to list all existing VMs.
```sh
crusoe compute vms list
```
**UI:**
To list VMs via the [console](https://console.crusoecloud.com), go to the [Instances](https://console.crusoecloud.com/compute/instances) page.
**Terraform:**
To list existing instances using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.
```hcl
# get instance with the specified ID
data "crusoe_compute_instance" "instance" {
id = ""
}
output "crusoe_instances" {
value = data.crusoe_compute_instance.instance
}
```
## Update an existing VM
Currently, you can start or stop an existing VM. You can also update properties like VM type, IP address type, and Infiniband partition.
A stopped VM retains all data stored on that VM. An on-demand stopped VM is not billed, but a VM currently in an instance commitment is billed regardless of its state. Learn more about [VM billing](./overview#vm-billing).
**CLI:**
Use the `compute vms ` command to change the state of an already existing VM.
Use the `compute vms update ` command to make updates to the Infiniband partition, VM type, or public IP type via the `--ib-partition-id`, `--type`, or `--public-ip-type` flags, respectively.
**UI:**
To update the state of a VM via the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Navigate to the row of the VM you want to update.
3. Under actions, select **Start** or **Stop**.
To update properties of a VM via the [console](https://console.crusoecloud.com):
1. Click on the name of the VM you want to update. This opens its details page.
2. Change its properties using the dropdowns or buttons.
**Terraform:**
To update an existing instance using the Crusoe Terraform provider, you can change the fields of an existing instance resource and run `terraform apply`. The Crusoe Terraform provider will apply the changes to the instance.
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
// new VM
resource "crusoe_compute_instance" "my_vm" {
name = "my-vm"
type = "l40s-48gb.1x"
location = "us-southcentral1-a"
image = "ubuntu22.04:latest" # use the 'latest' flag to get the most up to date image available from Crusoe
ssh_key = local.ssh_key
disks = [
{
id = crusoe_storage_disk.data_disk.id
mode = "read-only"
attachment_type = "data"
}
]
}
```
Currently, only the "network_interfaces" and "disks" of the instance can be changed. Changes to other fields of the instance will force a re-creation of the instance (deletion and then creation of a new instance).
### Updating state from within the VM
You can use standard `unix` commands like `shutdown` and `reboot` to change the state of a running VM. If you run `shutdown`, the VM will shut down and transition to the `Stopped` state. If you run `reboot`, the VM will reboot and will remain unreachable until it has fully restarted; however, the state will still be shown as `Running` throughout the restart.
## Deleting a VM
:::info
**Warning:** deleting a VM will also delete all data stored on the VM. Do not delete a VM unless you also wish to delete any downloaded or derived data.
:::
**CLI:**
Use the `compute vms delete` command to delete a VM of your choice. As an example, you can delete a VM by replacing `VM_NAME` with the name of the VM you wish to delete:
```sh
crusoe compute vms delete VM_NAME
```
**UI:**
To delete a VM via the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Click on the VM you want to delete. This opens its details page.
3. Select **Delete Instance**.
4. Input the name of the VM in the input box and click **Confirm**.
**Terraform:**
A VM can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
If you are having issues creating or deleting VMs, please [contact support](../../resources/support.md).
---
# Accessing your VMs
## SSH into a VM
SSH is the primary method of accessing VMs. When you create a VM, you will be given a public IP addres by default (e.g. `91.106.222.0`) as well as a private IP address (e.g. `172.27.0.10`) to access the newly created VM, either over the public internet or within your VPC. You can access these IPs via the instance page in the console, or with `crusoe compute vms get `.
You should ssh in as the [default user](../images/overview#default-user) for your operating system. Also, make sure your local machine has the same SSH key you used when [creating the VM](./managing-ssh-keys.mdx).
```
ssh ubuntu@91.106.222.0
```
By default, Public IPs are tied to the lifecycle of the VM, and will change if a VM is stopped and started. You can optionally enable a static public IP, or disable public IP assignment entirely.
Private IPs are static, and will not change based on the lifecycle of the VM.
### Firewall Rules
VPC networks contain [default firewall rules](../../networking/firewall-rules/overview#default-firewall-rules) that allow for SSH access to all VMs from the public internet, as well as other VMs on the subnet. If this is removed, SSH access will be removed.
If you are unable to `ssh` into a VM, ensure that your firewall rules allow `ssh` (destination: `tcp:22`) access.
### Handling errors
Occasionally there will be an error with the SSH connection. If you see the `Network unreachable` or `Connection refused` errors, especially immediately after starting a VM, please wait a minute or two and try again. If you see other errors and are unable to log in to a running virtual machine, please [contact support](../../resources/support.md).
### Setting longer SSH timeouts
If you want to set a longer timeout, you can modify your `/etc/ssh/sshd_config`, for example to keep a connection alive for 24 hours:
```
ClientAliveInterval 120
ClientAliveCountMax 720
```
Once you've saved the file, restart the `ssh` service by running `service sshd restart`. Note that if you're sshed in, you will get disconnected.
### Add additional SSH keys to a VM
If you wish to allow other users access to your VM, you can add their SSH public key to the `~/.ssh/authorized_keys` file on the VM you wish to grant them access to, replacing `SSH_PUBLIC_KEY` with the desired public key:
```
echo "SSH_PUBLIC_KEY" >> ~/.ssh/authorized_keys
```
The newly added user should now be able to SSH into the machine their key was added to using the steps outlined above.
## Internal DNS
You can also reach VMs within the VPC using the internal DNS address: `$VM_NAME.$LOCATION.compute.internal`. Learn more at [Internal DNS](/networking/vpc-networks/overview#internal-dns).
## Serial Console access
In the event that you are unable to SSH into your VM (e.g. the SSH server is not running or your keys are not present on the VM), or there is a problem with your VM, you can use the serial console to access the VM's serial port.
### Enabling and disabling serial console access
In order to enable serial console access, you will need to set an appropriate user and password by establishing an SSH connection and using `passwd` to set a password:
```sh
# Set a local password for the currently logged in user
sudo passwd $(whoami)
```
### Using the serial console
The serial console is available via the CLI, using the `compute vms serial-console` command:
```sh
crusoe compute vms serial-console \
--name NAME \
--port-num PORT
```
This will open a connection to the serial console on the VM.
#### Exiting the serial console
You can exit the serial console by typing `~.` on a newline.
#### Port number
Port number must be between 1 to 4 inclusive, with a maximum of one connection to a port at a time.
- Port 1 _(default, recommended)_: The standard console for Ubuntu. Use this for all login and boot troubleshooting.
- Ports 2-4: Reserved for custom kernel debugging. These ports will not provide a login prompt on standard Crusoe images.
#### Serial console timeout
The serial console times out after 30 minutes, regardless of activity (or inactivity).
---
# Managing lifecycle scripts
Lifecycle scripts allow you to react to VM state changes, either a VM starting up or a VM shutting down, by running a shell script.
Startup scripts are run on VM boot after the network is accessible, and can run for an indefinite amount of time. Shutdown scripts are run when the VM gets a [ACPI shutdown signal](https://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface), and will run for up to 90 seconds, after which the VM will be terminated. Scripts run as `root` and execute in the root directory (`/`).
## Creating lifecycle scripts
In order to create a lifecycle script, you need to create a VM with the script(s):
**CLI:**
Use the `vm create` command to create a VM of your choice and pass in the path to a local script
```sh
crusoe compute vms create \
--name my-vm \
--type l40s-48gb.1x \
--keyfile ~/.ssh/id_ed25519.pub \
--startup-script ~/path/to/startup.sh \
--shutdown-script ~/path/to/shutdown.sh
```
**UI:**
To create a VM with Lifecycle Scripts via the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Click **Create Instance** and begin the instance creation flow.
3. When you reach the **Lifecycle Scripts** step, enter your startup and/or shutdown scripts. These are saved in `/usr/local/bin/crusoe` on the VM.
4. Click **Create Instance**.
## Modifying lifecycle scripts
You can modify an existing lifecycle script directly on the VM by editing the files in `/usr/local/bin/crusoe`. Changes take effect immediately on the next shutdown or startup.
## Limitations
Lifecycle scripts are subject to the following limitations:
- Scripts must be in `bash` (has to start with `#!/bin/bash`)
- Scripts must be less than 64kB in size
- Scripts may only contain ASCII characters
### Example
```
"startup_script": "#!/bin/bash\necho 'hello'"
```
---
# VM Telemetry
Command Center provides you visibility into the health and performance of your Crusoe virtual machines. Two categories of telemetry are available for VMs:
**Metrics:** Infrastructure metrics covering GPU, CPU, memory, disk, and network performance, collected at 60-second intervals and retained for 30 days. Available for NVIDIA GPU, AMD GPU, and non-GPU VMs. A subset is viewable in the Console; the full dataset is available via Prometheus-compatible API and Grafana.
In the Console, navigate to **Compute**, select your VM, then select the **Metrics** tab.
**Logs:** JournalD system logs are collected from each VM and available to search, filter, and query. Supported for NVIDIA GPU accelerated instances, AMD GPU accelerated instances, and non-GPU instances. Logs are retained for 7 days. In the Console, navigate to **Managed Logs** in the left navigation bar to search across all VMs, or navigate to **Compute**, select your VM, then select the **Logs** tab.
:::note
Custom metrics (application-defined metrics from your workloads) are available for CMK clusters only. You can't use custom metrics for standalone VMs.
:::
For installation, token generation, and access method details, see [Get started](../../command-center/get-started.mdx). VM telemetry requires agent version **vm-v1.0.3 or higher**.
## VM Instance Actions
After you install the Crusoe Watch Agent, you can perform the following actions from your VM instance page in the Crusoe Console:
- **Generate bug report** — Create an NVIDIA or AMD bug report for VMs with NVIDIA or AMD GPUs. Download it or attach it to a support ticket for troubleshooting GPU-related issues.
- **Report an issue** — Open a pre-filled Zendesk support ticket with VM information and attach the latest bug report if applicable.
To access these features:
1. Navigate to [**Compute**](https://console.crusoecloud.com/compute/instances) in the left navigation bar.
2. Select your VM from the list.
3. Click on the three vertical dots icon next to the Start/Stop VM button and then click **Generate bug report** or **Report issues**.
You can also generate and download bug reports with the CLI and API. For requirements, collection steps, and error messages, see [Diagnostics](../../command-center/diagnostics.mdx).
## NVIDIA GPU Metrics
The following metrics are available for VMs with NVIDIA GPU instances:
| **Metrics** | **Definition** | **Suggested Query** |
| :------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| TFlops (FP16) | The measured 16-bit floating-point GPU throughput calculated by scaling the tensor core utilization against the hardware's theoretical maximum. | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE / 100 \* theoretical max TFLOPS |
| GPU Utilization (%) | The percentage of time the GPU is actively executing tasks. | DCGM_FI_DEV_GPU_UTIL |
| CPU Utilization (%) | The aggregated percentage of time the host's CPU cores are busy over the last 60 seconds. | (sum without(cpu, mode) (rate(crusoe_vm_cpu_seconds_total\{vm_id="vm-id", mode!="idle"\}[60s]))) / (sum without(cpu, mode) (rate(crusoe_vm_cpu_seconds_total\{vm_id="vm-id"\}[60s]))) \* 100 |
| GPU Memory Utilization (%) | The percentage of total dedicated GPU memory that is actively allocated and consumed by processes on the GPU. | (DCGM_FI_DEV_FB_USED / ( DCGM_FI_DEV_FB_FREE + DCGM_FI_DEV_FB_USED)) \* 100 |
| GPU Memory Bandwidth Utilization (%) | The percentage of the theoretical peak memory interface bandwidth being utilized for data transfer between the GPU and memory. | DCGM_FI_PROF_DRAM_ACTIVE |
| System Memory Utilization (%) | The percentage of total host system RAM that is consumed. | (crusoe_vm_memory_used_bytes / crusoe_vm_memory_total_bytes) \* 100 |
| GPU Power Draw (W) | The current power consumption of the GPU, measured in Watts. | DCGM_FI_DEV_POWER_USAGE |
| GPU Temperature (Celsius) | The current core temperature of the GPU die, measured in Celsius. | DCGM_FI_DEV_GPU_TEMP |
| Tensor Core Utilization (%) | The percentage of time the Tensor pipeline is actively processing instructions over the sample period. | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE \* 100 |
| VPC Network Bandwidth In (bytes per second) | The rate of data received by the host machine via the VPC network interface, measured in bytes per second. | crusoe_vm_network_receive_bytes_total |
| VPC Network Bandwidth Out (bytes per second) | The rate of data transmitted by the host machine via the VPC network interface, measured in bytes per second. | crusoe_vm_network_transmit_bytes_total |
| PCIe Bandwidth (bytes per second) | The rate of data transfer (Tx + Rx) between the CPU host memory and the GPU over the PCIe bus, measured in bytes per second. | DCGM_FI_PROF_PCIE_TX_BYTES + DCGM_FI_PROF_PCIE_RX_BYTES |
| PCIe Replay Rate | The rate of error-induced packet retransmissions over the PCIe bus, measured in replays per second. High rates indicate link quality issues. | rate(DCGM_FI_DEV_PCIE_REPLAY_COUNTER[1m]) |
| Uncorrectable ECC Error Rate | The rate of accumulation of uncorrectable double-bit memory errors (DBE) on the GPU, indicating severe hardware instability. | rate(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1m]) |
| Correctable ECC Error Rate | The rate of accumulation of correctable single-bit memory errors (SBE) on the GPU, indicating marginal hardware stability. | rate(DCGM_FI_DEV_ECC_SBE_VOL_TOTAL[1m]) |
| SM Occupancy (%) | The average percentage of available resident warps running concurrently on the Streaming Multiprocessors (SMs) over the sample period. | DCGM_FI_PROF_SM_OCCUPANCY |
| SM Active (%) | The percentage of time the Streaming Multiprocessors (SMs) were executing instructions during the sample period. | DCGM_FI_PROF_SM_ACTIVE \* 100 |
| SM Average Clock Speed (MHz) | The current instantaneous clock frequency of the GPU's Streaming Multiprocessors (SMs) in Megahertz (MHz). | DCGM_FI_DEV_SM_CLOCK |
| GPU XID error | The most recent unique error code emitted by the GPU driver (a hardware or software fault ID). Non-zero values indicate an error that typically requires a driver reset or GPU restart. | DCGM_FI_DEV_XID_ERRORS |
### NVIDIA NVLink Metrics
If your VM uses NVLink-enabled NVIDIA instances, the following additional NVLink metrics are available:
| **Metrics** | **Definition** | **Suggested Query** |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| GPU NVLink Bandwidth In (bytes per second) | The rate of data received by the GPU from other GPUs over all active NVLink connections, measured in bytes per second. | DCGM_FI_DEV_NVLINK_RX_BYTES |
| GPU NVLink Bandwidth Out (bytes per second) | The rate of data transmitted by the GPU to other GPUs over all active NVLink connections, measured in bytes per second. | DCGM_FI_DEV_NVLINK_TX_BYTES |
## AMD GPU Metrics
:::note
AMD GPU Metrics is currently in preview and available for AMD MI300x and MI355x instances. Please reach out to [Crusoe Cloud Support](https://support.crusoecloud.com/hc/en-us/requests/new) to learn more.
:::
If your VM uses AMD GPU instances, the following AMD-specific metrics are supported:
| **Metrics** | **Definition** | **Suggested Query** |
| :---------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------ |
| GPU Memory Utilization (%) | The percentage of total dedicated GPU memory that is actively allocated and consumed by processes on the GPU. | (gpu_used_visible_vram / gpu_total_visible_vram) \* 100 |
| GPU Utilization (%) | The percentage of time the GPU graphics engine is actively executing tasks. | gpu_gfx_activity |
| GPU Power Draw (W) | The current power consumption of the GPU, measured in Watts. Requires AMD Device Metrics Exporter v1.5.0 or later. | gpu_power_usage |
| GPU Memory Bandwidth Utilization (%) | The percentage of the theoretical peak memory interface bandwidth being utilized for data transfer between the GPU and memory. | gpu_umc_activity |
| PCIe Bandwidth (bytes per second) | The rate of data transfer (Tx + Rx) between the CPU host memory and the GPU over the PCIe bus, measured in bytes per second. | pcie_bandwidth |
| GPU Temperature (Celsius) | The current junction temperature of the GPU, measured in Celsius. | gpu_junction_temperature |
| GPU XGMI Bandwidth In (bytes per second) | The rate of data received by the GPU from other GPUs over XGMI (AMD's high-speed interconnect), measured in bytes per second over the last minute. | rate(gpu_xgmi_link_rx[1m]) |
| GPU XGMI Bandwidth Out (bytes per second) | The rate of data transmitted by the GPU to other GPUs over XGMI (AMD's high-speed interconnect), measured in bytes per second over the last minute. | rate(gpu_xgmi_link_tx[1m]) |
| PCIe Replay Rate (replays per second) | The rate of error-induced packet retransmissions over the PCIe bus, measured in replays per second over the last minute. High rates indicate link quality issues. | rate(pcie_replay_count[1m]) |
| Uncorrectable ECC Error Rate | The rate of accumulation of uncorrectable ECC memory errors on the GPU over the last minute, indicating severe hardware instability. | rate(gpu_ecc_uncorrect_total[1m]) |
| Correctable ECC Error Rate | The rate of accumulation of correctable ECC memory errors on the GPU over the last minute, indicating marginal hardware stability. | rate(gpu_ecc_correct_total[1m]) |
:::note
JournalD system logs are also collected for AMD GPU accelerated instances, the same as for NVIDIA GPU instances. Access them via Managed Logs in the Console or API.
:::
## CPU-Only Instance Metrics
The following metrics are available for CPU-only VMs (non-GPU instances):
| **Metrics** | **Definition** | **Suggested Query** |
| :------------------------------------------- | :------------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CPU Utilization (%) | The aggregated percentage of time the host's CPU cores are busy over the last 60 seconds. | (sum without(cpu, mode) (rate(crusoe_vm_cpu_seconds_total\{vm_id="vm-id", mode!="idle"\}[60s]))) / (sum without(cpu, mode) (rate(crusoe_vm_cpu_seconds_total\{vm_id="vm-id"\}[60s]))) \* 100 |
| System Memory Utilization (%) | The percentage of total host system RAM that is consumed. | (crusoe_vm_memory_used_bytes / crusoe_vm_memory_total_bytes) \* 100 |
| VPC Network Bandwidth In (bytes per second) | The rate of data received by the host machine via the VPC network interface, measured in bytes per second. | crusoe_vm_network_receive_bytes_total |
| VPC Network Bandwidth Out (bytes per second) | The rate of data transmitted by the host machine via the VPC network interface, measured in bytes per second. | crusoe_vm_network_transmit_bytes_total |
:::note
CPU, System Memory, and VPC Network metrics are also available for GPU-accelerated instances (both NVIDIA and AMD) in addition to their GPU-specific metrics.
:::
## Disk Metrics
The following metrics reflect boot disk and persistent disk activity, as observed from the VM. Ephemeral disk activity isn't included; see [NVMe Drive Health Metrics](#nvme-drive-health-metrics) below for ephemeral disk health metrics.
| **Metrics** | **Definition** | **Suggested Query** |
| :-------------------------------------- | :------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------- |
| Disk Read IOPS | The rate of disk read operations completed per second. | `rate(crusoe_vm_disk_reads_completed_total[60s])` |
| Disk Write IOPS | The rate of disk write operations completed per second. | `rate(crusoe_vm_disk_writes_completed_total[60s])` |
| Disk Read Bandwidth (bytes per second) | The rate of data read from disk. | `rate(crusoe_vm_disk_read_bytes_total[60s])` |
| Disk Write Bandwidth (bytes per second) | The rate of data written to disk. | `rate(crusoe_vm_disk_write_bytes_total[60s])` |
| Disk Read Latency (p99) | The 99th percentile latency of disk read operations. | `histogram_quantile(0.99, rate(crusoe_vm_disk_read_latency_seconds_bucket[60s]))` |
| Disk Write Latency (p99) | The 99th percentile latency of disk write operations. | `histogram_quantile(0.99, rate(crusoe_vm_disk_write_latency_seconds_bucket[60s]))` |
| Disk Space Used (%) | The percentage of total disk capacity currently in use, per mounted filesystem. | `(crusoe_vm_disk_bytes_used / crusoe_vm_disk_bytes_total) * 100` |
| Disk Inode Usage (%) | The percentage of total inodes currently in use, per mounted filesystem. | `(crusoe_vm_disk_inodes_used / crusoe_vm_disk_inodes_total) * 100` |
## NVMe Drive Health Metrics
For VMs with [Ephemeral Disks](../../storage/disks/overview.md#ephemeral-disks) (local NVMe drives), the following SMART/health metrics are available:
NVMe drive health metrics are supported for `s1a` instances and all GPU instances except L40S.
| **Metrics** | **Definition** | **Suggested Query** |
| :----------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------- |
| NVMe Critical Warning | Indicates whether a SMART critical warning bit is set (spare capacity low, temperature, reliability, read-only, volatile memory backup failed, or PMR unreliable). | `crusoe_vm_nvme_smart_critical_warning == 1` |
| NVMe Media Errors | The cumulative count of uncorrectable media and data integrity errors. | `rate(crusoe_vm_nvme_media_errors_total[1h])` |
| NVMe Percentage Used | The percentage of the drive's rated endurance consumed (100 indicates rated endurance has been reached). | `crusoe_vm_nvme_percentage_used >= 90` |
| NVMe Available Spare (%) | The remaining spare capacity available on the drive. | `crusoe_vm_nvme_available_spare` |
| NVMe Power-On Hours | The lifetime number of hours the drive has been powered on. | `crusoe_vm_nvme_power_on_hours` |
## Object Storage VM Metrics
The following metrics track this VM's connections to Object Storage endpoints:
| **Metrics** | **Definition** | **Suggested Query** |
| :------------------------------ | :--------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------- |
| Object Store Connections | Total connection phases observed to the Object Storage endpoint. | `rate(crusoe_vm_objectstore_connections_total[5m])` |
| Object Store Connection Latency | Average connection-phase latency to the Object Storage endpoint. | `rate(crusoe_vm_objectstore_connection_latency_seconds[5m]) / rate(crusoe_vm_objectstore_connections_total[5m])` |
| Object Store TCP Retransmits | TCP retransmissions to the Object Storage endpoint. | `rate(crusoe_vm_objectstore_tcp_retransmits_total[5m])` |
| Object Store Bytes Sent | Total bytes sent to the Object Storage endpoint. | `rate(crusoe_vm_objectstore_bytes_sent_total[5m])` |
| Object Store Bytes Received | Total bytes received from the Object Storage endpoint. | `rate(crusoe_vm_objectstore_bytes_recv_total[5m])` |
:::note
These metrics report aggregate connection-phase statistics per endpoint rather than per-request (GET/PUT) latency. TLS prevents distinguishing individual requests at the TCP connection layer, so per-request latency requires a proxy-based measurement approach instead.
:::
These are VM-side metrics only; Object Storage doesn't currently expose bucket-level metrics. See [Object Storage Overview](../../storage/object-storage/overview.md) for more on Object Storage itself.
:::note
Disk, NVMe drive health, and Object Storage VM metrics are currently available only via the API, Telemetry Conduit, and Crusoe MCP. Console support is planned for a future release.
:::
For token generation and querying metrics via API or Grafana, see [Get started](../../command-center/get-started.mdx) and [Metrics](../../command-center/metrics.md).
## Considerations
### Correctable ECC errors not emitted correctly for multi-GPU VMs and clusters
Correctable ECC errors (`DCGM_FI_DEV_ECC_SBE_VOL_TOTAL`) may not be emitted with all timeseries for multi-GPU VMs and clusters by the DCGM exporter due to a known NVIDIA NVLink metrics bug ([GitHub Issue](https://github.com/NVIDIA/dcgm-exporter/issues/581)). We are working on a long-term fix. In the interim, you can manually force DCGM to start monitoring this field by running the following command within the VM:
```sh
dcgmi dmon -e 310
```
### Clusters with Slurm images not retrieving metrics correctly due to pre-installed dcgm-exporter
If you have a pre-installed dcgm-exporter systemd service, it could conflict with the dcgm-exporter that would be installed as part of installing the Crusoe Watch Agent, causing metrics collection failures. To prevent this issue, use `--replace-dcgm-exporter` to replace your existing dcgm-exporter with the Crusoe version for full metrics collection. The `service_name` is an optional field that defaults to `dcgm-exporter.service`.
```sh
sudo crusoe-watch-agent --replace-dcgm-exporter [SERVICE_NAME]
```
---
# Optimizing GPU performance
Across all of the NVIDIA-based Crusoe accelerated instances it is possible to further optimize the performance of the
GPUs by leveraging a clock locking mechanism to reduce latency and maximize performance of the workloads. With the NVIDIA
driver installed apply the following settings:
Ensure that the GPUs are in persistent mode:
```
sudo nvidia-smi -pm 1
```
**Graphics Clock Locking** - To reduce clock switching latency and ensure that the maximum SM clock is available across all
execution kernels in your code. You can set all GPUs to the maximum value:
```
nvidia-smi -i 0 --query-supported-clocks="gr" --format=csv,noheader | head -n 1 | awk '{print $1}' | xargs sudo nvidia-smi -lgc
```
**Memory Clock Locking** - To reduce clock switching latency and ensure that the maximum memory clock is available across all
execution kernels and DMAs in yoru code. You can set all GPUs to maximum value:
```
nvidia-smi -i 0 --query-supported-clocks="mem" --format=csv,noheader | head -n 1 | awk '{print $1}' | xargs sudo nvidia-smi -lmc
```
You can confirm that the settings were applied based on the clock SMs/Memory values below:
| GPU | Max Graphics Clock (Mhz) | Max Memory Clock (MHz) |
| --------------------- | ------------------------ | ---------------------- |
| NVIDIA A100 80GB PCIe | 1410 | 1512\* |
| NVIDIA A100 80GB SXM4 | 1410 | 1593 |
| NVIDIA H100 SXM5 | 1980 | 2619 |
\*Setting the memory clock through the locking mechanism is not supported
For workloads that can tolerate memory errors (ie. Graphics targeted workloads) or if your code has an out-of-band error
correcting mechanism. You can maximize performance of your code by disabling ECC. To disable ECC run the following:
```
sudo nvidia-smi -e 0
```
which will disable ECC for all GPUs in the instance, a reboot of the instance will be required to take effect.
**MIG** - To partition certain GPU types into multiple instances for running different, isolated workloads, you can run
```
sudo nvidia-smi -i 0 -mig 1
# -i 0 partitions gpu with index 0
```
For more information and to see which Nvidia GPUs support this, see this [link](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html#abstract).
---
# Burn-in validation
Every Crusoe node runs up to 30 validation tests before it becomes available for you to use. Tests cover three domains—**node**, **GPU**, and **fabric**—and follow this escalating pattern: sanity → validation → performance → stress.
If a node fails a test, it doesn't enter the fleet. Crusoe triages, repairs or replaces, and re-validates nodes before deployment.
## Test domains
| DOMAIN | APPLIES TO | PURPOSE |
| :------------------------------ | :------------------------ | :------------------------------------------------ |
| [Node](#node-tests) | All nodes | CPU, memory, storage, networking, and PCIe. |
| [NVIDIA GPU](#nvidia-gpu-tests) | NVIDIA nodes | DCGM diagnostics, ECC, PCIe link, and power. |
| [AMD GPU](#amd-gpu-tests) | AMD nodes | AGFHC diagnostics, HBM, compute, and performance. |
| [Fabric](#fabric-tests) | InfiniBand and RoCE nodes | RDMA bandwidth and collective communications. |
## Node tests
The following tests are run on every node regardless of GPU configuration:
### Sanity
| TEST | WHAT IT CHECKS |
| :---------------------------- | :--------------------------------------------------------------------------- |
| `node/global/sanity/boot` | Node boots successfully; basic system services are running. |
| `node/global/sanity/pcie` | Node's expected GPUs, NICs, and NVMe controllers are enumerated and visible. |
| `node/global/sanity/registry` | Node internet connectivity is established and validated. |
### Validation
| TEST | WHAT IT CHECKS |
| :--------------------------- | :---------------------------------------------------------------------------------- |
| `node/global/validation/nic` | Node's ethernet interface configuration, link state, and driver settings validated. |
### Performance
| TEST | WHAT IT CHECKS |
| :---------------------------------------- | :---------------------------------------------- |
| `node/global/perf/cpu` | CPU throughput (sysbench). |
| `node/global/perf/memory` | Host memory bandwidth (sysbench). |
| `node/global/perf/storage-block-bw` | Block storage sequential read/write throughput. |
| `node/global/perf/storage-block-iops` | Block storage random IOPS. |
| `node/global/perf/storage-block-lat` | Block storage I/O latency. |
| `node/global/perf/storage-ephemeral-bw` | Local NVMe sequential read/write throughput. |
| `node/global/perf/storage-ephemeral-iops` | Local NVMe random IOPS. |
| `node/global/perf/storage-ephemeral-lat` | Local NVMe I/O latency. |
### Stress
| TEST | WHAT IT CHECKS |
| :----------------------- | :------------------------------------------------------- |
| `node/global/stress/cpu` | Sustained CPU stability and thermal behavior under load. |
## NVIDIA GPU tests
The following tests are run on all NVIDIA GPU nodes:
| TEST | WHAT IT CHECKS |
| :------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| `gpu/nv/sanity/health` | GPU health: GPU presence, driver health, ECC errors, thermal throttling events, and prior XID errors utilizing DCGMI level 2. |
| `gpu/nv/validation/health` | GPU memory, compute, and subsystem validation utilizing DCGMI level 4. |
| `gpu/nv/validation/ecc` | Validate ECC is enabled with no uncorrectable errors. |
| `gpu/nv/validation/pcie` | PCIe link width and speed match rated values for the SKU type; any link below spec is flagged critical. |
| `gpu/nv/validation/power` | GPU reaches and sustains rated TDP under continuous compute load (DCGMI). |
| `gpu/nv/stress/power` | GPU stability under pulsed load (20 ramp-up/ramp-down cycles); catches marginal power delivery hardware errors that steady-state testing misses. |
## AMD GPU tests
The following tests are run on all AMD GPU nodes:
| TEST | WHAT IT CHECKS |
| :------------------------- | :--------------------------------------------------------------------------------------------------- |
| `gpu/amd/validation/agfhc` | AGFHC Level 5—full-node scan of compute units, HBM controllers, and memory subsystem. |
| `gpu/amd/stress/hbm` | HBM sustained bandwidth; catches memory controller issues that survive compute-only stress tests. |
| `gpu/amd/stress/compute` | Sustained arithmetic load across AMD GPUs. |
| `gpu/amd/stress/full` | Combined GPU test: combined compute, memory, and thermal load on a single GPU. |
| `system/amd/stress/mixed` | Mixed workload across GPU, CPU, and host memory simultaneously; matches real training load profiles. |
| `gpu/amd/stress/hpl` | ROCm HPL benchmark—exercises full HIP stack, memory bandwidth, and interconnect. |
## Fabric tests
The following tests are run on all InfiniBand and RoCE-enabled nodes:
### Global (all fabric nodes)
| TEST | WHAT IT CHECKS |
| :---------------------------- | :--------------------------------------------------------------------------------------------------- |
| `fabric/global/sanity/link` | InfiniBand or RoCE links are up and not in a degraded state. |
| `fabric/global/perf/ethernet` | Validates NIC and switch fabric: Frontend Ethernet throughput between node pairs (utilizing iperf3). |
### NVIDIA InfiniBand
The following tests are layered to isolate local hardware failures from fabric failures:
| TEST | WHAT IT CHECKS |
| :---------------------------- | :-------------------------------------------------------------------------- |
| `fabric/nv/perf/nvlink` | Single node: NVLink bandwidth using local IP as RDMA target. |
| `fabric/nv/perf/gpu-p2p` | Single node: GPU-to-GPU bandwidth using NCCL on a single node. |
| `fabric/nv/perf/rdma-bw` | Multi-node: InfiniBand fabric bandwidth, error rates, and GPU-Direct RDMA1. |
| `fabric/nv/stress/collective` | Multi-node: NCCL all-reduce over InfiniBand backend network. |
### AMD RoCE
| TEST | WHAT IT CHECKS |
| :----------------------------- | :--------------------------------------------------------------------------------------------------------------------------- |
| `fabric/amd/perf/rdma-bw` | Multi-node: GPU-Direct RDMA bandwidth across all 8 NICs per node; includes fault attribution to avoid pulling healthy nodes. |
| `fabric/amd/stress/collective` | Multi-node: RCCL all-reduce over RoCE—required to pass before distributed training is considered fleet-ready. |
## Cluster handoff
Beyond automated burn-in, Crusoe validates cluster deployments with real distributed workloads before handoff, running multi-node PyTorch jobs to confirm the full software stack behaves correctly under customer-representative load.
## Additional resources
- [Crusoe Blog](https://www.crusoe.ai/blog)
- [NVIDIA DCGM diagnostics](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/dcgm-diagnostics.html)
- [AMD AGFHC](https://instinct.docs.amd.com/projects/gpu-operator/en/latest/test/agfhc.html)
---
# Overview
Instance Templates offer an easy way to save commonly-used VM configurations across your organization or project. Templates may then be used to atomically create multiple VMs that are identically configured.
## How Instance Templates work
You can create one or more templates in your project that allow you to specify and save the following attributes associated with VM creation.
| Field Name | Description |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Instance Type` | The type of instance you want to create (e.g.: l40s-48gb.10x, h100-80gb-sxm-ib.8x, etc.). |
| `Location` | The location where your VMs must be created (e.g.: us-east1-a). You may also choose to create a 'Global' template. Global templates allow you to re-use the same template across different locations. Region-specific attributes, like InfiniBand network IDs and location must be specified during VM creation for global templates. |
| `Image Name and Version` | The image name and version to be used for VMs created from the template (e.g.: ubuntu22.04-nvidia-pcie-docker:latest) |
| `Disks to Create` | Templates allow you to create one or more [persistent disks](../../storage/disks/managing-persistent-disks.mdx) of specified sizes that will be attached to VMs created from the template. Currently, attaching existing disks is not supported via templates. |
| `VPC Network and Subnet` | The VPC network and subnet to place VMs in. For global templates, this defaults to the default VPC and subnet in the region selected during VM creation from the template. |
| `InfiniBand Network and Partition` | The IB network or partition to place VMs in. Must be specified during VM creation for global templates. Only required for IB-enabled VMs. |
| `Public SSH Key` | The SSH key used to connect to VMs. |
| `Lifecycle Scripts ` | Startup and Shutdown scripts that will be applied to VMs created from the template. |
| `Reservation ID ` | The ID of the reservation to be used when creating a VM using the template. |
---
# Managing Templates
## Create a new Template
**CLI:**
Use the `compute templates create` command to create a template:
```sh
crusoe compute templates create \
--name myfirsttemplate \
--type c1a.8x \
--location us-east1-a \
--image ubuntu22.04:latest \
--vpc-subnet-id 7a2f1b6a-9e48-4f3c-a0d7-1d9a8c2f3e0b \
--keyfile ~/.ssh/id_ed25519.pub
```
**UI:**
To create a template using the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instance Templates](https://console.crusoecloud.com/compute/instance-templates)** in the left nav.
2. Click **Create Template**.
3. Follow the UI flow to input all required elements.
4. Click **Create**.
## Creating VMs from Templates
VM creation from a template uses the [Bulk VM create method](https://docs.crusoecloud.com/api/#tag/VMs/operation/createInstance) under the hood. This means that the desired number of VMs are created atomically, with automated rollbacks in the event of any instance creation failures.
VMs (and disks created and attached to VMs) also inherit their naming from the template name by default. For example, VMs created from a template called 'test' will be named 'test-1', 'test-2', etc., with disks qualified with a further -disk prefix ('test-disk-1'). This may be overriden during individual creation requests.
**CLI:**
Use the `compute vms bulk-create` command, passing a `template-id` field associated with the chosen template:
```sh
crusoe compute vms bulk-create \
--template-id 9b6f2a8c-4e39-1d7a-0c8f-3e0a9b2c8f3e \
--count 3
```
If you are using a global template, you may optionally choose to specify a VPC subnet ID. Additionally, the InfiniBand network and partition IDs must be specified for IB-enabled instances provisioned via a template.
**UI:**
To create VMs from an available instance template:
1. From the console, select **Compute** > **[Instance Templates](https://console.crusoecloud.com/compute/instance-templates)** in the left nav.
2. Find the template you want to use and click **Create VMs**.
3. Specify the number of instances you want to create along with any other required fields based on your template type.
4. Click **Create**.
## Updating Templates
Templates once created cannot be modified. You may choose to clone your templates to make changes or delete outdated / unused templates.
## Deleting Templates
**CLI:**
Use the `compute vms delete` command, passing the name of the template to be deleted.
```sh
crusoe compute templates delete --name TEMPLATE_NAME
```
**UI:**
To delete an instance template:
1. From the console, select **Compute** > **[Instance Templates](https://console.crusoecloud.com/compute/instance-templates)** in the left nav.
2. Find the template you want to delete and select the **Delete** icon.
3. Confirm the deletion by typing in the name.
---
# Overview
## Images
Crusoe Cloud provides two types of VM images: Curated images, which are created and maintained by the Crusoe Cloud team, and Custom images, which are customized by the developer for use within their organization's project based on the curated image.
### Curated Images
Curated images are images provided by Crusoe Cloud and made available to all developers on the platform. These images contain a base operating system, e.g. Ubuntu, and additional software where specified, to provide a "batteries included" experience.
#### List of Curated Images
Here is a list of current images
| Image Name | Description | Lifecycle Stage |
| -------------------------------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `ubuntu22.04` | Ubuntu 22.04 base image. | GA |
| `ubuntu22.04-nvidia-sxm-docker` | Ubuntu 22.04 with NVIDIA drivers supporting SXM type GPUs, InfiniBand support, and a docker runtime. | GA |
| `ubuntu22.04-nvidia-pcie-docker` | Ubuntu 22.04 with NVIDIA drivers supporting PCIe type GPUs and a docker runtime. | GA |
| `ubuntu22.04-nvidia-slurm` | Ubuntu 22.04 with NVIDIA drivers and Slurm support. | GA |
| `ubuntu-rocm` | Ubuntu 22.04 with AMD ROCm drivers supporting OAM type GPUs. | GA |
| `ubuntu20.04` | Ubuntu 20.04 base image. | [EOL (May 2025)](../../resources/deprecation_notices.md) |
| `ubuntu20.04-nvidia-sxm-docker` | Ubuntu 20.04 with NVIDIA drivers supporting SXM type GPUs, InfiniBand support, and a docker runtime. | [EOL (May 2025)](../../resources/deprecation_notices.md) |
| `ubuntu20.04-nvidia-pcie-docker` | Ubuntu 20.04 with NVIDIA drivers supporting PCIe type GPUs and a docker runtime. | [EOL (May 2025)](../../resources/deprecation_notices.md) |
Each image also has a `tag` which is commonly `latest` for the latest version or a `yyyy-mm-dd` type string for the date the image was created.
#### Default user
The default user for all `ubuntu` based images is `ubuntu`. `root` access is considered to be deprecated, and you will be warned on login.
### Custom Images
Custom images are customer-managed, modified versions of curated images. Once customers have created a virtual machine (VM) from a base image and customized it with preferred configurations and software, they can stop the VM and create a custom image from its boot disk.
When creating subsequent VMs within their project, customers can select their newly created custom image as the base image. Unlike curated images, custom images are region and project specific and cannot be shared outside the project or the specific region it was created in.
Custom images can only be created from the boot disks of a VM. Like curated images, custom images support tags with the default `tag` being `latest`.
#### Custom Image maintenance
Customers are solely responsible for the maintenance of their custom images. Crusoe does not update, manage or modify those images in any way and cannot guarantee or troubleshoot performance on custom images. Deleting the source VM used to create the custom image is allowed and will not affect the custom image itself, though this can only be done after the custom image creation operation is completed successfully.
Please remove any local disk mount entries from the /etc/fstab file on the VM before creating a custom image in order to prevent boot failures for subsequent VMs.
:::info
**Note**: While customers can upgrade the kernel version on their custom images, it can lead to issues with driver and other software compatibility on Crusoe Cloud. Please [contact support](../../resources/support.md) before performing major kernel version upgrades.
:::
#### Quotas
By default, customers can create up to 128 custom images per project. To request an increase in this quota, please [contact support](../../resources/support.md)
#### Limitations
Customers cannot do the following:
1. Create a custom image from a VM while it is not SHUTOFF.
2. Start a VM while a custom image is being created from it.
3. Create a custom image from a disk which is not an OS disk.
4. Delete all tags for a custom image. Every custom image must have at least one tag.
#### Billing
Custom images are billed based on the amount of data used in the persistent disk. Refer persistent disk pricing [here](../../storage/disks/overview.md)
---
# Managing Images
## Create Custom Images
**CLI:**
Use the `compute images create` to create a custom image:
```sh
crusoe compute images create \
--name \
--vm-name \
--tags \
--description \
--project-name
```
**UI:**
To create a new custom image via the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Select the instance you want to use for the custom image (make sure it's powered off).
3. Click **Create a custom image**.
4. Enter the custom image name, description, and tags and click **Create**.
You can also create a custom image from the [Custom Images](https://console.crusoecloud.com/compute/images) page in the console:
1. From the console, select **Compute** > **[Custom Images](https://console.crusoecloud.com/compute/images)** in the left nav.
2. Click **Create Custom Image**.
3. Enter a name and description, and select your source VM. If you don't see the VM you want to use, power it off and retry the operation.
:::info
The Create Custom Image workflow can take a few minutes to complete and the time taken depends on the used capacity of the source VM OS disk. **Do not turn the source VM on while it is being created.**
:::
## Listing Images
**CLI:**
Use the `compute images list` to list all images:
```sh
crusoe compute images list
```
**UI:**
You can view custom images in the [Custom Images](https://console.crusoecloud.com/compute/images) page.
**Terraform:**
```hcl
# Data source to get all custom images
data "crusoe_compute_custom_image" "clean_list_images" {
}
output "custom_image_list" {
value = data.crusoe_compute_custom_image.clean_list_images.custom_images
}
```
## Using Images
**CLI:**
Use the `compute vms create` command, passing an `image` flag with the chosen curated image or passing a `--custom-image` flag with the chosen custom image:
```sh
crusoe compute vms create \
--image ubuntu:22.04 \
...
crusoe compute vms create \
--custom-image my-custom-image \
...
```
**UI:**
To use a VM image in the console:
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Click **Create Instance**.
3. Select one of the **Instance Options**:
- **Default**: Enables you to create an instance with a curated image.
- **Custom**: Enables you to create an instance with a custom image.
4. Use the **Select an image** and **Select an image version** dropdowns to select an image and version.
5. Continue with the remainder of the **Create Instance** flow.
**Terraform:**
```hcl
# Create the VM using the custom image
resource "crusoe_compute_instance" "my_vm_with_custom_image" {
name = "my-vm"
location = "us-east1-a"
type = "c1a.2x"
custom_image = "my-custom-image" # Use custom_image parameter with image name
ssh_key = file("~/.ssh/id_ed25519.pub")
}
```
## Update Custom Images
**CLI:**
Use the `compute images update` command, passing an `description` flag to update the description:
```sh
crusoe compute images update \
--description my-new-description \
...
```
Use the `compute images add-tags` command, passing an `tags` flag to add tags to the custom image:
```sh
crusoe compute images add-tags \
--tags my-new-tag1,my-new-tag2 \
...
```
Use the `compute images delete-tags` command, passing an `tags` flag to delete tags from the custom image:
```sh
crusoe compute images delete-tags \
--tags my-tags-to-delete1,my-tags-to-delete2 \
...
```
**UI:**
To update a custom image in the console:
1. From the console, select **Compute** > **[Custom Images](https://console.crusoecloud.com/compute/images)** in the left nav.
2. Find the custom image you want to edit and click the edit icon.
3. Edit the tags and description as necessary.
4. Click **Update** to apply the changes.
## Delete Custom Images
**CLI:**
Use the `compute images delete` command:
```sh
crusoe compute images delete
```
**UI:**
To delete a custom image in the console:
1. From the console, select **Compute** > **[Custom Images](https://console.crusoecloud.com/compute/images)** in the left nav.
2. Find the custom image you want to delete.
3. Click the delete icon in the row for that image.
4. Input the name of the custom image in the input box and click **Confirm**.
---
# Object storage overview
Crusoe Cloud Object Storage provides high-performance, S3-compatible object storage designed for AI/ML workloads. Store and retrieve datasets, model checkpoints, training artifacts, and other unstructured data with the standard S3 format. It is ideal for petabyte-scale datasets and can be used to migrate data from other cloud providers to Crusoe.
## Key Features
- **S3-Compatible API**: Use existing S3 tools and libraries (boto3, s3cmd, rclone, aws s3 cli) without modification
- **High Performance**: Optimized for large file uploads and downloads common in ML workflows
- **Regional Storage**: Data stored in the same location as your VMs for low-latency access
- **Versioning & Object Lock**: Protect critical data from accidental deletion or modification
- **Multipart Upload Support**: Efficient handling of large files with automatic chunking
- **Pre-Signed URL**: Direct access to a private Objects without exposing your credentials
## Prerequisites
Before using Object Storage, ensure the following:
1. You have an active Crusoe Cloud Organization with Object Storage enabled. Contact your account team or Crusoe support if you do not see Object Storage in your Console.
2. Your VMs are running in a location where the Object Storage bucket is created.
## Architecture
Object Storage is a **regional resource** - buckets are created in specific locations and can only be accessed from VMs in the same location. This design ensures low-latency access and high throughput for data-intensive workloads.
### Object Storage Endpoints
Each location has a dedicated Object Storage endpoint:
```
https://object..crusoecloudcompute.com
```
For example:
- `https://object.us-east1-a.crusoecloudcompute.com`
- `https://object.us-southcentral1-a.crusoecloudcompute.com`
### Authentication
Object Storage uses dedicated Object Storage API keys (access key and secret key pairs) for authentication. These are separate from your Crusoe Cloud API tokens and are managed through the Console or CLI. See [Managing Object Storage API Keys](./managing-storage-api-keys.mdx) for more information.
## Naming Rules
### Bucket Names
Bucket names must:
- Be unique across a Crusoe Cloud region
- Be between 3 and 63 characters long
- Contain only lowercase letters, numbers, and hyphens
- Start and end with a letter or number
- Not contain consecutive hyphens
- Not be formatted as an IP address (e.g., 192.168.1.1)
### Object Storage Keys
Object Storage uses dedicated AWS S3 style API keys (an access key and secret key pair) for authentication. These are separate from your Crusoe Cloud API tokens and are required for all Object Storage client operations. Each bucket user can have a maximum of 2 keys, similar to AWS S3 key restrictions. Object Storage access keys can only be viewed by the owner of the key.
## Getting Started
1. **Create an Object Storage API Key**: Generate access credentials for Object Storage
2. **Create a Bucket**: Set up an Object Storage bucket in your desired location
3. **Configure Your Object Storage Client**: Point your tools to the Crusoe Object Storage endpoint
4. **Upload and Download Objects**: Use standard S3 operations to manage your data
See [Managing Object Storage API Keys](./managing-storage-api-keys.mdx) and [Managing Buckets](./managing-buckets.mdx) for detailed instructions.
## Supported S3 Features
Crusoe Object Storage supports the following S3 features:
- Basic object operations (PUT, GET, DELETE, HEAD)
- Multipart uploads
- Bucket and object listing
- Bucket versioning
- Object locking (WORM - Write Once Read Many)
- Bucket tagging
- Object metadata
- Range requests (partial downloads)
- Presigned URLs
Features not currently supported:
- Server-side encryption (SSE)
- Access Control Lists (ACLs) beyond bucket-level permissions
- Cross-region replication
- Lifecycle policies
- Event notifications
## Performance Characteristics
- **Upload Speed**: Optimized for large file uploads (64 MB+ objects recommended)
- **Download Speed**: High-throughput reads for training pipelines
- **Multipart Upload**: Automatic chunking for large files
- **Concurrency**: High concurrent request handling for distributed workloads
For optimal performance:
- Use multipart uploads for files larger than 64 MB
- Increase concurrency settings in your S3 client
- Ensure your VM type has sufficient VPC network bandwidth
## Billing
Object storage is priced at $0.06 per GiB per month, billed based on the average amount of data stored over the billing period.
### Pricing Unit
Object Storage is priced per GiB per month. Storage usage is measured in binary gibibytes (GiB), where 1 GiB = 230 bytes (1,073,741,824 bytes). Similarly, 1 TiB = 240 bytes, or 1,024 GiB.
### Usage Calculation
Object Storage usage is billed based on the amount of data stored in your object store, measured over time. Crusoe samples your Object Storage usage at regular intervals throughout each hour and computes the average usage (in GiB) for that hour. This average represents your consumption for that hour, expressed in GiB-Hours (GiB-Hr). Your total monthly usage is the sum of all hourly averages across the billing period. The monthly invoice reflects the total GiB-Hr consumed and the corresponding cost.
For example, suppose your object store holds 100 GiB for the first hour and 150 GiB for the second hour. Your usage for those two hours is:
(100 GiB × 1 hr) + (150 GiB × 1 hr) = 250 GiB-Hr. To estimate an equivalent monthly rate from a quoted $/GiB/month price, divide by the number of hours in the month (for example, 730 for a 30-day month).
### Billing Period
Storage is billed monthly. There are no minimum storage duration requirements and no early deletion fees. You pay only for what you store, for as long as you store it.
### Restrictions
1. Buckets are private by default and can only be accessed by the owner of the bucket.
2. Object Storage endpoints are not reachable from the public internet.
3. Only path style URLs are supported, no virtual hosted style URLs.
4. Object Storage API keys cannot be rotated in place. Customers must create a new key and delete the old one.
5. Storage tiering is not supported currently in Crusoe Cloud Object Storage
6. Once versioning is enabled on a bucket, it cannot be disabled.
## Next Steps
- [Managing Object Storage API Keys](./managing-storage-api-keys.mdx) - Create and manage authentication credentials
- [Managing Buckets](./managing-buckets.mdx) - Create and configure storage buckets
- [Using S3 Tools](./using-s3-tools.md) - Configure s3cmd, rclone, boto3, and other clients
- [Troubleshooting](./troubleshooting.md) - Common issues and solutions
For VM-side connection latency, throughput, and reliability metrics for Object Storage traffic, see [VM Telemetry](../../compute/virtual-machines/vm-telemetry.md#object-storage-vm-metrics).
---
# Managing Crusoe Cloud Object Storage API Keys
# Managing Object Storage API Keys
:::info Credential and resource scoping
Object Storage API keys are user-scoped — only the user who created a key can view or manage it. You can use object storage API keys in all buckets in all projects within your organization. Buckets are project-scoped resources.
:::
## Creating an Object Storage API Key
**CLI:**
Use the `storage tokens create` command to generate a new Object Storage API key:
```sh
crusoe storage tokens create --alias my-training-key
```
Optional parameters:
- `--alias ` — A human-readable name for the key.
- `--expires-at ` — Expiration date for the key in RFC3339 format (e.g., `2021-12-03T19:58:34Z`).
The command outputs an access key ID and a secret key. **Save the secret key immediately** — it cannot be retrieved again after creation.
Example output:
```
Access Key ID: CKIAXXXXXXXXXXXXXXXX
Secret Key: SKXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Alias: my-training-key
Expires At: 2026-12-31T00:00:00Z
```
**UI:**
1. From the [console](https://console.crusoecloud.com), click **Admin** in the bottom-left corner.
2. Select **Security** > **[Object Storage Keys](https://console.crusoecloud.com/security/object-storage-keys)** in the left nav.
3. Click **Create Object Storage Key**.
4. Enter an alias for the key and optionally set an expiration date.
5. Click **Create**.
6. Copy and securely store the access key and secret key. The secret key is shown only once.
**Terraform:**
Creating a storage API key is required to authenticate with the S3-compatible API for Object Storage in Crusoe Cloud. The following is intended to help get you started in using Terraform to provision a storage API key.
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates a storage API key:
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_storage_s3_key" "my_key" {
alias = "my-storage-key"
}
output "access_key_id" {
value = crusoe_storage_s3_key.my_key.access_key_id
}
output "secret_access_key" {
value = crusoe_storage_s3_key.my_key.secret_access_key
sensitive = true
}
```
`alias` is an optional argument that provides a human-readable name for the key.
`expire_at` is an optional argument that specifies an expiration date for the key in RFC 3339 format (e.g., `2025-12-31T23:59:59Z`).
:::warning
The `secret_access_key` is only available at creation time. Make sure to save the output of `terraform apply` as the secret cannot be retrieved later.
:::
After saving the code to a `main.tf` file, the following commands serve as the process to create a resource in Crusoe Cloud using Terraform:
`terraform init` - Initializes a working directory containing Terraform configuration files.
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will create the resources.
To view the secret access key after creation, run `terraform output secret_access_key`.
## Listing Object Storage API Keys
**CLI:**
```sh
crusoe storage tokens list
```
**UI:**
1. From the [console](https://console.crusoecloud.com), click **Admin** in the bottom-left corner.
2. Select **Security** > **[Object Storage Keys](https://console.crusoecloud.com/security/object-storage-keys)** in the left nav.
3. All active keys are listed with their alias, access key ID, created date, and expiration date.
**Terraform:**
To list existing storage API keys using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.
```hcl
# list storage API keys
data "crusoe_storage_s3_keys" "keys" {}
output "crusoe_storage_keys" {
value = data.crusoe_storage_s3_keys.keys
}
```
## Deleting an Object Storage API Key
**CLI:**
```sh
crusoe storage tokens delete
```
Replace `` with the access key of the key you wish to delete. For example, if the access key is `CKIAXGFV74Z2FFURA9UA`, then the command would be
```sh
crusoe storage tokens delete CKIAXGFV74Z2FFURA9UA
```
The access key can also be obtained using the list [../list/_cli.mdx] command.
**UI:**
1. From the [console](https://console.crusoecloud.com), click **Admin** in the bottom-left corner.
2. Select **Security** > **[Object Storage Keys](https://console.crusoecloud.com/security/object-storage-keys)** in the left nav.
3. Click the delete icon next to the Object Storage API key you wish to remove.
4. Confirm the deletion.
**Terraform:**
A storage API key can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
:::danger Warning
Deleting an Object Storage API key immediately revokes access for any clients configured with that key.
:::
---
# Managing Buckets
Buckets are created and managed through the Crusoe Cloud Console or CLI. They cannot be created or deleted through S3 client tools.
Bucket names must comply with the [naming rules](./overview#naming-rules) described in the Overview, including being globally unique across a Crusoe Cloud region and between 3–63 characters using only lowercase letters, numbers, and hyphens.
By default the buckets are only accessible by bucket owner via up to 2 Object Storage API keys.
## Creating a Bucket
**CLI:**
```sh
crusoe storage buckets create \
--name my-training-data \
--location us-east1-a
```
`--name` and `--location` are required. The bucket must be in the same location as the VMs that will access it.
**UI:**
1. Visit the [Crusoe Cloud Console](https://console.crusoecloud.com).
2. Navigate to **Storage** > **Buckets** in the left navigation.
3. Click **Create Bucket**.
4. Enter a name for the bucket.
5. Select the location.
6. (Optional)Enable versioning and object lock if desired.
7. Click **Create**.
**Terraform:**
Creating an Object Storage bucket is the first step to storing objects in Crusoe Cloud. The following is intended to help get you started in using Terraform to provision an S3-compatible bucket in Crusoe Cloud.
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates a bucket:
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_storage_s3_bucket" "my_bucket" {
name = "my-bucket"
location = "us-southcentral1-a"
}
```
`name` and `location` are required arguments.
`name` is a globally unique name for the bucket.
`location` is the Crusoe Cloud location where the bucket will be created.
After saving the code to a `main.tf` file, the following commands serve as the process to create a resource in Crusoe Cloud using Terraform:
`terraform init` - Initializes a working directory containing Terraform configuration files.
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will create the resources.
## Listing Buckets
**CLI:**
```sh
crusoe storage buckets list
```
Optional filters:
- `--location ` — Filter by location.
- `--tag key=,value=` — Filter by tag.
**UI:**
1. Navigate to **Storage** > **Buckets** in the Console.
2. All buckets in your project are listed with their name, location, size, and creation date.
**Terraform:**
To list existing buckets using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.
```hcl
# list buckets
data "crusoe_storage_s3_buckets" "buckets" {}
output "crusoe_buckets" {
value = data.crusoe_storage_s3_buckets.buckets
}
```
## Getting Bucket Details
**CLI:**
```sh
crusoe storage buckets get my-training-data
```
This returns the bucket's metadata including versioning state, object lock configuration, tags, and used capacity.
**UI:**
1. Navigate to **Storage** > **Buckets** in the Console.
2. Click on a bucket name to view its details, including the S3 endpoint, Bucket URL, versioning state, object lock configuration, tags, and current size.
**Terraform:**
To view the details of a specific bucket using Terraform, filter the `crusoe_storage_s3_buckets` data source by bucket name.
```hcl
data "crusoe_storage_s3_buckets" "all" {}
output "bucket_details" {
value = one([
for bucket in data.crusoe_storage_s3_buckets.all.buckets :
bucket if bucket.name == "my-training-data"
])
}
```
This returns the bucket's metadata including versioning state, object lock configuration, tags, and used capacity.
## Managing Bucket Tags
**CLI:**
Add tags:
```sh
crusoe storage buckets add-tags my-training-data \
--tag key=environment,value=production \
--tag key=team,value=ml-infra
```
Remove tags:
```sh
crusoe storage buckets remove-tags my-training-data \
--tag key=environment,value=production
```
List tags:
```sh
crusoe storage buckets list-tags my-training-data
```
**UI:**
1. Visit the [Crusoe Cloud Console](https://console.crusoecloud.com).
2. Navigate to **Storage** > **Buckets** in the left navigation.
3. Click on the bucket for which you want to add tags.
4. Click on the **Edit Bucket Details** button.
5. Enter the key-value pairs in the **Tags** section.
6. Click **Update**.
**Terraform:**
To manage tags on an existing bucket using the Crusoe Terraform provider, add or update the `tags` field on your existing `crusoe_storage_s3_bucket` resource and run `terraform apply`.
```hcl
resource "crusoe_storage_s3_bucket" "my_bucket" {
# ... existing bucket configuration ...
tags = {
environment = "production"
team = "ml"
}
}
```
To remove all tags, remove the `tags` field or set it to an empty map:
```hcl
resource "crusoe_storage_s3_bucket" "my_bucket" {
# ... existing bucket configuration ...
tags = {}
}
```
After making any changes, save the code and then perform the following commands:
`terraform plan` - the output of this command will show the planned changes to the bucket's tags.
`terraform apply` - this command will apply the changes.
## Enabling Versioning
Versioning preserves every version of every object in the bucket, protecting against accidental overwrites and deletions.
:::danger Warning
Once versioning is enabled, it cannot be disabled or suspended.
:::
**CLI:**
```sh
crusoe storage buckets enable-versioning my-training-data
```
**UI:**
1. Visit the [Crusoe Cloud Console](https://console.crusoecloud.com).
2. Navigate to **Storage** > **Buckets** in the left navigation.
3. Click on the bucket for which you want to Enable Versioning.
4. Click on the **Edit Bucket Details** button.
5. Click on the **Versioning** radio button.
6. Click **Update**.
**Terraform:**
To enable versioning on a bucket using the Crusoe Terraform provider, set the `versioning_enabled` field to `true` on a `crusoe_storage_s3_bucket` resource and run `terraform apply`.
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_storage_s3_bucket" "my_bucket" {
name = "my-bucket"
location = "us-southcentral1-a"
versioning_enabled = true
}
```
:::warning
Enabling versioning is irreversible. Once versioning is enabled on a bucket, it cannot be disabled.
:::
After making any changes, save the code and then perform the following commands:
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will apply the changes.
## Enabling Object Lock
Object lock protects objects from being deleted or overwritten for a specified retention period. Enabling object lock **automatically** enables versioning.
:::danger Warning
Once object lock is enabled on a bucket, it cannot be disabled, and versioning cannot be suspended.
:::
**CLI:**
```sh
crusoe storage buckets enable-locking my-training-data \
--retention 30d
```
The `--retention` flag accepts values like `7d` (7 days) or `1y` (1 year).
**UI:**
1. Visit the [Crusoe Cloud Console](https://console.crusoecloud.com).
2. Navigate to **Storage** > **Buckets** in the left navigation.
3. Click on the bucket for which you want to Enable Object Lock.
4. Click on the **Versioning** radio button.
5. Click on the **Object Lock** radio button.
6. Click **Update**.
**Terraform:**
To enable object locking on a bucket using the Crusoe Terraform provider, set `versioning_enabled`, `object_lock_enabled`, `retention_period`, and `retention_period_unit` on a `crusoe_storage_s3_bucket` resource and run `terraform apply`.
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_storage_s3_bucket" "my_bucket" {
name = "my-bucket"
location = "us-southcentral1-a"
versioning_enabled = true
object_lock_enabled = true
retention_period = 30
retention_period_unit = "days"
}
```
`versioning_enabled` must be set to `true` because object locking requires versioning.
`object_lock_enabled` enables object locking on the bucket.
`retention_period` is the duration for which objects are locked.
`retention_period_unit` is the unit for the retention period (e.g., `days`).
:::warning
Enabling object locking is irreversible. Both versioning and object locking cannot be disabled once enabled.
:::
After making any changes, save the code and then perform the following commands:
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will apply the changes.
## Deleting a Bucket
:::danger Warning
A bucket must be empty before it can be deleted. Deleting a bucket is a permanent action.
:::
**CLI:**
```sh
crusoe storage buckets delete --name my-training-data
```
**UI:**
1. Navigate to **Storage** > **Buckets** in the Console.
2. Click the delete icon next to the bucket you wish to delete.
3. Enter the name of the bucket to confirm and click **Delete**
**Terraform:**
A bucket can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
---
# Using S3 Tools
Once you have an Object Storage API key and at least one bucket, you can use standard S3-compatible tools to manage objects. All S3 operations are performed directly from your Crusoe Cloud VMs.
## S3 Endpoint
Use the following endpoint format for your location:
```
https://object..crusoecloudcompute.com
```
For example:
```
https://object.us-east1-a.crusoecloudcompute.com
```
---
## Configuring s3cmd
[s3cmd](https://s3tools.org/s3cmd) is a command-line tool for interacting with S3-compatible storage.
### Installation
```sh
# Ubuntu/Debian
sudo apt-get install s3cmd
```
### Configuration
Create or edit `~/.s3cfg` with the following:
```ini
[default]
access_key = YOUR_ACCESS_KEY
secret_key = YOUR_SECRET_KEY
host_base = object..crusoecloudcompute.com
host_bucket = object..crusoecloudcompute.com
use_https = True
signature_v2 = False
```
:::important
Set `host_bucket` to the same value as `host_base` (without a `%(bucket)s` prefix) because Crusoe Object Storage uses path-style URLs only.
:::
Alternatively, run `s3cmd --configure` and manually set the endpoint values.
### Common Operations
```sh
# List all buckets
s3cmd ls
# List objects in a bucket
s3cmd ls s3://my-training-data
# Upload a file
s3cmd put model-checkpoint.tar s3://my-training-data/checkpoints/
# Upload a directory recursively
s3cmd put --recursive ./dataset/ s3://my-training-data/datasets/
# Download a file
s3cmd get s3://my-training-data/checkpoints/model-checkpoint.tar ./
# Download a directory recursively
s3cmd get --recursive s3://my-training-data/datasets/ ./local-datasets/
# Delete an object
s3cmd del s3://my-training-data/checkpoints/old-checkpoint.tar
# Get object info (metadata)
s3cmd info s3://my-training-data/checkpoints/model-checkpoint.tar
# Multipart upload (automatic for files > 15 MB)
# Adjust chunk size if needed:
s3cmd put --multipart-chunk-size-mb=64 large-dataset.tar s3://my-training-data/
# List active multipart uploads
s3cmd multipart s3://my-training-data
```
---
## Configuring rclone
[rclone](https://rclone.org/) is a versatile tool for managing files on cloud storage, and is particularly useful for syncing data and migrating objects from other cloud providers into Crusoe.
### Installation
```sh
# Ubuntu/Debian
sudo apt-get install rclone
```
### Configuration
Run `rclone config` and create a new remote, or manually add the following to `~/.config/rclone/rclone.conf`:
```ini
[crusoe]
type = s3
provider = Other
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = https://object..crusoecloudcompute.com
acl = private
force_path_style = true
```
:::important
The `force_path_style = true` setting is required because Crusoe Object Storage does not support virtual-hosted-style URLs.
:::
### Common Operations
```sh
# List all buckets
rclone lsd crusoe:
# List objects in a bucket
rclone ls crusoe:my-training-data
# Upload a file
rclone copy ./model-checkpoint.tar crusoe:my-training-data/checkpoints/
# Upload a directory
rclone copy ./dataset/ crusoe:my-training-data/datasets/
# Download a file
rclone copy crusoe:my-training-data/checkpoints/model-checkpoint.tar ./
# Sync a local directory to a bucket (mirror)
rclone sync ./dataset/ crusoe:my-training-data/datasets/
# Check data integrity
rclone check ./dataset/ crusoe:my-training-data/datasets/
# Get file info
rclone lsl crusoe:my-training-data/checkpoints/
```
### Migrating Data from AWS S3
rclone can transfer data directly between cloud providers. To copy data from AWS S3 into Crusoe Object Storage:
1. Configure an AWS S3 remote in rclone (named `aws` in this example).
2. Run:
```sh
rclone copy aws:source-bucket/path/ crusoe:my-training-data/path/ \
--transfers 16 \
--checkers 8 \
--s3-upload-concurrency 4
```
Adjust `--transfers` and related flags based on your available bandwidth and the number of files.
---
## Configuring boto3 (Python)
[boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) is the AWS SDK for Python, widely used in ML pipelines and data processing scripts.
### Installation
```sh
sudo apt install python3-boto3
```
### Configuration
```python
import boto3
s3 = boto3.client(
"s3",
endpoint_url="https://object..crusoecloudcompute.com",
aws_access_key_id="YOUR_ACCESS_KEY",
aws_secret_access_key="YOUR_SECRET_KEY",
)
```
:::note
The `region_name` parameter is not required. If your S3 client requires one, you can set it to any placeholder value (e.g., `us-east-1`). The Crusoe S3 endpoint handles routing internally.
:::
### Common Operations
```python
# List buckets
response = s3.list_buckets()
for bucket in response["Buckets"]:
print(bucket["Name"])
# List objects in a bucket
response = s3.list_objects_v2(Bucket="my-training-data")
for obj in response.get("Contents", []):
print(obj["Key"], obj["Size"])
# Upload a file
s3.upload_file(
"model-checkpoint.tar",
"my-training-data",
"checkpoints/model-checkpoint.tar",
)
# Upload with multipart (automatic for large files)
from boto3.s3.transfer import TransferConfig
config = TransferConfig(
multipart_threshold=64 * 1024 * 1024, # 64 MB
multipart_chunksize=64 * 1024 * 1024,
max_concurrency=10,
)
s3.upload_file(
"large-dataset.tar",
"my-training-data",
"datasets/large-dataset.tar",
Config=config,
)
# Download a file
s3.download_file(
"my-training-data",
"checkpoints/model-checkpoint.tar",
"./model-checkpoint.tar",
)
# Delete an object
s3.delete_object(
Bucket="my-training-data",
Key="checkpoints/old-checkpoint.tar",
)
# Get object metadata
response = s3.head_object(
Bucket="my-training-data",
Key="checkpoints/model-checkpoint.tar",
)
print(f"Size: {response['ContentLength']}, Last Modified: {response['LastModified']}")
# Copy an object within the same bucket
s3.copy_object(
Bucket="my-training-data",
Key="checkpoints/model-checkpoint-backup.tar",
CopySource="my-training-data/checkpoints/model-checkpoint.tar",
)
```
### Using boto3 with a Session
For scripts that interact with multiple buckets or need credential management:
```python
import boto3
session = boto3.Session(
aws_access_key_id="YOUR_ACCESS_KEY",
aws_secret_access_key="YOUR_SECRET_KEY",
)
s3 = session.resource(
"s3",
endpoint_url="https://object..crusoecloudcompute.com",
)
# Upload using the resource interface
bucket = s3.Bucket("my-training-data")
bucket.upload_file("local-file.bin", "remote-key/local-file.bin")
# Iterate all objects
for obj in bucket.objects.all():
print(obj.key, obj.size)
```
---
## Benchmarking Object Storage
You can use standard S3 benchmarking tools to measure performance from within your Crusoe Cloud VMs. Below is an example using [elbencho](https://github.com/breuner/elbencho), a distributed storage benchmark for file systems, object stores, and block devices.
### Configure S3 Credentials
Before running benchmarks, configure your S3 credentials using one of the methods below. Elbencho reads credentials from the standard AWS credential chain, so there is no need to pass keys directly on the command line.
#### Option A: AWS Config File (Recommended)
Using an AWS config file avoids exposing credentials in shell history.
```sh
# Create the AWS config directory if it does not exist
mkdir -p ~/.aws
```
Add your Crusoe Object Storage credentials to `~/.aws/credentials`:
```ini
[crusoe]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
```
Then export the profile and bucket name:
```sh
export AWS_PROFILE=crusoe
export AWS_ENDPOINT_URL_S3="https://object..crusoecloudcompute.com"
export BUCKET_NAME="YOUR_BUCKET_NAME"
```
#### Option B: Environment Variables
If you need to override credentials (for example, in CI pipelines), export them directly:
```sh
export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
export AWS_ENDPOINT_URL_S3="https://object..crusoecloudcompute.com"
export BUCKET_NAME="YOUR_BUCKET_NAME"
```
:::important
Replace `` with your Crusoe Cloud location (e.g., `us-east1-a`). Create a dedicated bucket for benchmarking to avoid impacting production data.
:::
### Install elbencho
Download the latest static binary from the [releases page](https://github.com/breuner/elbencho/releases). The static executable includes S3 support and has no external dependencies.
```sh
# Detect architecture, download, and extract
ARCH=$(uname -m | sed 's/arm64/aarch64/')
wget https://github.com/breuner/elbencho/releases/latest/download/elbencho-static-${ARCH}.tar.gz
tar -xf elbencho-static-${ARCH}.tar.gz
sudo mv elbencho /usr/local/bin/elbencho
```
### Run a Write Benchmark
This writes 10 objects of 1 GiB each across 1 directory using 128 threads:
```sh
elbencho \
--s3endpoints $AWS_ENDPOINT_URL_S3 \
--s3objprefix ${HOSTNAME}/ \
--write \
--size 1G \
--block 16M \
--dirs 1 \
--files 10 \
--threads 128 \
$BUCKET_NAME
```
| Flag | Description |
| :----------------------------------- | :------------------------------------------------------- |
| `--s3endpoints $AWS_ENDPOINT_URL_S3` | Crusoe S3 endpoint URL |
| `--s3objprefix ${HOSTNAME}/` | Per-host prefix, enables multi-VM runs without conflicts |
| `--write` | Write mode |
| `--size 1G` | Object size (1 GiB per object) |
| `--block 16M` | Block size per request (16 MiB) |
| `--dirs 1` | Number of directories |
| `--files 10` | Number of objects per directory |
| `--threads 128` | Number of concurrent threads |
| `--lat` | Report latency statistics |
### Run a Read Benchmark
After writing objects, run a read benchmark against the same data:
```sh
elbencho \
--s3endpoints $AWS_ENDPOINT_URL_S3 \
--s3objprefix ${HOSTNAME}/ \
--read \
--size 1G \
--block 16M \
--dirs 1 \
--files 10 \
--threads 128 \
$BUCKET_NAME
```
Use the same `--size`, `--files`, `--dirs`, and `--s3objprefix` values as the write benchmark to ensure the read test targets the same objects.
### Run a Mixed Workload Benchmark
To run a combined write and read benchmark:
```sh
elbencho \
--s3endpoints $AWS_ENDPOINT_URL_S3 \
--s3objprefix ${HOSTNAME}/ \
--write --read \
--size 64M \
--block 1M \
--dirs 1 \
--files 256 \
--threads 16 \
$BUCKET_NAME
```
This runs both write and read phases sequentially against the same set of objects, giving you throughput numbers for both operations in a single run.
### Save Results to a File
Use the `--resfile` and `--csvfile` flags to persist benchmark results:
```sh
elbencho \
--s3endpoints $AWS_ENDPOINT_URL_S3 \
--s3objprefix ${HOSTNAME}/ \
--write \
--size 1G \
--block 16M \
--dirs 1 \
--files 10 \
--threads 128 \
--lat --nolive \
--resfile results.txt \
--csvfile results.csv \
$BUCKET_NAME
```
### Clean Up Benchmark Objects
After benchmarking, remove the test objects:
```sh
elbencho \
--s3endpoints $AWS_ENDPOINT_URL_S3 \
--s3objprefix ${HOSTNAME}/ \
--dirs 1 \
--files 10 \
--threads 128 \
--delfiles \
--s3multidel 1 \
$BUCKET_NAME
```
Verify that the bucket is empty:
```sh
aws s3 ls s3://$BUCKET_NAME --recursive
aws s3 rm s3://$BUCKET/${HOSTNAME}/ --recursive
```
To remove the bucket, either use the Crusoe Cloud Console as described [here](./managing-buckets.mdx#deleting-a-bucket). If you enabled versioning, you will need to delete all versions of the objects before deleting the bucket.
---
# Troubleshooting object storage
This guide covers common issues you may encounter when using Crusoe Cloud Object Storage and their solutions.
## "403 Access Denied" when using S3 tools
**Possible causes and solutions:**
- **Incorrect credentials**: Verify your access key and secret key are correct and have not expired.
- **Region mismatch**: Confirm the endpoint URL matches the **location** of your bucket. Object Storage is a regional resource — you cannot access buckets in one location from VMs in another.
- **Path-style configuration**: Check that `host_bucket` (s3cmd) or `force_path_style` (rclone/boto3) is configured correctly for path-style access.
**Example fix for s3cmd:**
```ini
# In ~/.s3cfg, ensure these are set correctly:
host_base = object..crusoecloudcompute.com
host_bucket = object..crusoecloudcompute.com # No %(bucket)s prefix
```
**Example fix for rclone:**
```ini
# In ~/.config/rclone/rclone.conf:
force_path_style = true
```
---
## "Bucket already exists" error on creation
**Cause**: Bucket names must be globally unique across all Crusoe Cloud projects.
**Solution**: Choose a different name or check for existing buckets with `crusoe storage buckets list`.
**Best practice**: Use a naming convention that includes your organization or project name:
- `mycompany-training-data-prod`
- `project123-ml-checkpoints`
---
## Slow upload/download performance
**Possible causes and solutions:**
### Use multipart uploads for large files
Most S3 clients automatically use multipart uploads for files larger than 15 MB, but you can adjust settings for better performance:
**s3cmd:**
```sh
s3cmd put --multipart-chunk-size-mb=64 large-file.tar s3://bucket/
```
**boto3:**
```python
from boto3.s3.transfer import TransferConfig
config = TransferConfig(
multipart_threshold=64 * 1024 * 1024,
multipart_chunksize=64 * 1024 * 1024,
max_concurrency=10,
)
s3.upload_file("large-file.tar", "bucket", "key", Config=config)
```
### Increase concurrency settings
**rclone:**
```sh
rclone copy ./data/ crusoe:bucket/path/ \
--transfers 16 \
--checkers 8 \
--s3-upload-concurrency 4
```
### Check VM network bandwidth
Verify your VM type has sufficient [VPC Network Bandwidth](https://docs.crusoecloud.com/compute/virtual-machines/overview). Larger instance types have higher network throughput.
### Use optimal object sizes
For best throughput, use object sizes of 64 MB or larger. If you're uploading many small files, consider creating tar archives before uploading.
---
## Cannot create or delete buckets via S3 clients
**Cause**: This is expected behavior. Bucket creation and deletion are managed exclusively through the Crusoe Cloud Console or CLI for security and resource management reasons.
**Solution**: Use the Crusoe CLI or Console:
```sh
# Create bucket
crusoe storage buckets create --name my-bucket --location us-east1-a
# Delete bucket
crusoe storage buckets delete --name my-bucket
```
---
## "Object lock requires versioning" error
**Cause**: Object lock can only be enabled on buckets that have versioning enabled.
**Solution**: Enable versioning first, then enable object lock:
```sh
# Enable versioning
crusoe storage buckets enable-versioning my-bucket
# Then enable object lock
crusoe storage buckets enable-locking my-bucket --retention 30d
```
:::danger Warning
Once versioning and object lock are enabled, they cannot be disabled.
:::
---
## S3 client reports "InvalidRequest" or "NotImplemented"
**Cause**: You're trying to use an S3 feature that isn't supported by Crusoe Object Storage.
**Currently unsupported features:**
- Server-side encryption (SSE-S3, SSE-KMS, SSE-C)
- Complex ACLs beyond bucket-level permissions
- Cross-region replication
- Event notifications (S3 Event Notifications to SNS/SQS)
**Solution**: Check the [Supported S3 Features](./overview.md#supported-s3-features) section in the Overview to confirm the feature is available.
---
## Multipart upload stuck or incomplete
**Symptoms**: Large file uploads fail or hang indefinitely.
**Causes and solutions:**
1. **Network interruption**: Check your VM's network connectivity and retry the upload.
2. **List incomplete uploads**:
```sh
# s3cmd
s3cmd multipart s3://bucket-name
# boto3
response = s3.list_multipart_uploads(Bucket='bucket-name')
```
3. **Abort incomplete uploads**:
```python
# Using boto3
s3.abort_multipart_upload(
Bucket='bucket-name',
Key='object-key',
UploadId='upload-id-from-list'
)
```
---
## Secret key not saved or lost
**Cause**: The secret key is only displayed once during creation and cannot be retrieved afterward.
**Solution**:
1. Delete the old S3 API key
2. Create a new S3 API key
3. Update your S3 client configuration with the new credentials
```sh
# Delete old key
crusoe storage tokens delete
# Create new key
crusoe storage tokens create --alias my-new-key
```
---
## "Connection timed out" or "Unable to connect"
**Possible causes:**
1. **Wrong endpoint**: Verify you're using the correct endpoint for your location:
```
https://object..crusoecloudcompute.com
```
2. **VM not in same location**: Object Storage is regional. Ensure your VM is in the same location as your bucket.
3. **Network issues**: Check your VM's network connectivity:
```sh
curl -I https://object..crusoecloudcompute.com
```
---
## Need more help?
If you're still experiencing issues after trying these solutions:
1. Check the Crusoe Cloud [status page](https://status.crusoecloud.com) for service incidents
2. Review the [Object Storage Overview](./overview.md) for architecture details
3. Contact [Crusoe Support](https://docs.crusoecloud.com/resources/support) with:
- Error messages (full text)
- S3 client configuration (redact credentials)
- Steps to reproduce the issue
- Your project ID and location
---
# Overview
Crusoe Cloud offers three types of disks: Ephemeral Disks, Persistent Disks and Shared Disks. Ephemeral Disks are local to the virtual machine (on the same physical server). Persistent Disks are remote block storage volumes that are attached to a virtual machine. Shared Disks are shared network filesystems, designed to be accessed from many virtual machines concurrently.
## Ephemeral Disks
Ephemeral Disks offer the highest performance with no additional redundancy. Ephemeral Disks have a lifecycle that is tied to the server associated with a given virtual machine. Local storage on [GPU](../../compute/virtual-machines/overview.md) and `s1a` instances are considered ephemeral disks. The disks are erased when the physical server reboots, or the VM is stopped and restarted, or if there are any other hardware or software failures. They are only suitable for use cases where data loss is tolerable, even if they are configured in a storage cluster like MinIO, Lustre or Ceph. If your data has a need to be protected, please use persistent or shared disks.
### Encryption
Ephemeral Disks use self-encrypting SSD drives that are local to each customer VM. Content is encrypted using AES-XTS. When a VM is destroyed, the content is cryptographically erased.
_Note: if a VM is restarted from within the VM (e.g. using `sudo reboot now`) rather than by stopping and restarting the VM from the UI, CLI, or API, the disks will not be erased._
## Persistent Disks
Persistent Disks offer reasonable performance with high availability and durability. Persistent Disks have a lifecycle independent of any given virtual machine, and can be attached or detached while the VM is running or stopped. Persistent Disks can be attached to a given VM then detached at a later date, therefore storing data while a VM is stopped.
You can create individual Persistent Disks ranging in size from **1 GiB** to **10 TiB** and you can attach up to **16** disks per instance, including the OS disk which is automatically attached. Disks are persistent NVMe, which will preserve data even after detaching the disk from the instance.
### Encryption
Persistent Disks are backed by volumes on a centralized storage cluster. Data is encrypted at rest, but the encryption key(s) are pooled among all users of that storage cluster. When a disk is deleted, data is soft-deleted, but cryptographic erasure is not currently available.
### OS Disks
VM operating systems are stored on a 128 GB Persistent Disk attached to a VM. OS disks are backed by volumes on a centralized storage cluster. Storage is encrypted at rest, and the encryption key is pooled among all storage users of that storage cluster.
## Shared Disks
Shared Disks offer a high level of scalability and performance and are attached to many VMs with shared access. Upon attaching a Shared Disk to one or more VMs, you can interface with the Shared Disk using standard NFS semantics. Like Persistent Disks, Shared Disks have a lifecycle independent of any given virtual machine, and can be attached or detached while the VM is running or stopped.
You can create Shared Disks ranging in size from **1 TiB** to **1000 TiB** in increments of 1 TiB and you can attach up to **8** disks per instance. Data stored on a Shared Disk will be preserved until explicitly deleted.
### Encryption
Shared Disks are backed by volumes on a centralized storage cluster. Data is encrypted at rest, but the encryption key(s) are pooled among all users of that storage cluster. Data is not encrypted while in transit. When a disk is deleted, data is soft-deleted, but cryptographic erasure is not currently available.
## Limitations
### Persistent Disks
- Can only be attached to a single instance at a time in `read-write` mode, but can be attached to multiple VMs in `read-only` mode. Multi-attach must be configured via the CLI or Terraform — the Console UI defaults to `read-write`.
- Can only be increased in size. Resizing cannot be done while the disk is attached to a VM.
- Must be unmounted before detaching. Detaching a mounted disk can cause VM crashes and loss of data on both the persistent and ephemeral disks.
- OS Disks cannot currently be resized.
### Shared Disks
- Names have a maximum length of 36 characters. Names longer than 36 characters cause the disk create operation to fail.
## Quotas
To view your disks quotas, see [Viewing Quotas](../../usage-billing/viewing-quotas.mdx).
## Billing
Ephemeral Disks are included in the VM pricing, and thus don't incur additional cost.
Shared Disks and Persistent Disks (including OS disks) are billed regardless of whether or not they are attached to a running VM.
| Type | Pricing |
| ----------------- | ---------------------------------------- |
| `persistent-disk` | $0.08/GiB/month for provisioned storage |
| `shared-disk` | $0.07/GiB/month for provisioned storage. |
## Next Steps
- [Managing Ephemeral Disks](./managing-ephemeral-disks.md)
- [Managing Persistent Disks](./managing-persistent-disks.mdx)
- [Managing Shared Disks](./managing-shared-disks.mdx)
- [Setting up the VAST NFS driver](./setup-nfs-driver.mdx)
- [Shared Disks Metrics](./shared-disks-metrics.mdx)
- [Troubleshooting](./troubleshooting.md)
## Related
- [Instance Templates](../../compute/instance-templates/overview.md) — provision VMs with disks attached at creation.
- [VM Images](../../compute/images/overview.md) — custom boot images backed by Persistent Disks.
- [Managed Kubernetes (CMK)](../../orchestration/cmk/cmk-addons.md) — use Persistent and Shared Disks as PersistentVolumes via the Crusoe CSI driver.
---
# Managing Ephemeral Disks
Ephemeral disks are available on certain VM instance types. Please refer to [VM overview page](../../compute/virtual-machines/overview.md) for details.
## Lifecycle
Ephemeral disks have no redundancy. Ephemeral disks have a lifecycle that is tied to the server associated with a given virtual machine. The disks are erased when the physical server reboots, or the VM is stopped and restarted, or if there are any other hardware or software failures causing the virtual machine to move or shut down. They are only suitable for use cases where data loss is tolerable, even if they are configured in a storage cluster like MinIO, Lustre or Ceph. If your data has a need to be protected, please back it up on an external disk like persistent or shared disk.
_Note: if a VM is restarted from within the VM (e.g. using `sudo reboot now`) rather than by stopping and restarting the VM from the UI, CLI, or API, the disks will not be erased._
## Formatting and Mounting Ephemeral Disks
Below is a script you can add to your [startup scripts](../../compute/virtual-machines/managing-lifecycle-scripts.mdx) to automatically detect all ephemeral disks on the VM, combine them into an unprotected `RAID0` array using `mdadm`, format the array with an `xfs` file system, and mount it at the `/raid0` path. The `RAID0` array is created even when only a single ephemeral disk is present so that the mount path is consistent regardless of disk count.
```sh
#!/bin/bash
set -euo pipefail
echo "info: detecting NVMe drives by-id..."
# Collect all nvme-* symlinks, exclude partitions
all_symlinks=$(ls -1 /dev/disk/by-id/nvme-* 2>/dev/null | grep -vE '(_[0-9]+$|part[0-9]+$)' || true)
# Deduplicate: keep only one symlink per backing device
nvme_devices=""
seen_targets=""
for symlink in $all_symlinks; do
target=$(readlink -f "$symlink")
if ! echo "$seen_targets" | grep -q -w "$target"; then
nvme_devices="$nvme_devices $symlink"
seen_targets="$seen_targets $target"
fi
done
nvme_devices=$(echo "$nvme_devices" | xargs) # trim
num_nvme=$(echo "$nvme_devices" | wc -w)
if [ "$num_nvme" -eq 0 ]; then
echo "error: no NVMe drives were detected under /dev/disk/by-id/. Exiting."
exit 1
fi
echo "info: found $num_nvme NVMe drive(s)."
echo "info: devices: $nvme_devices"
if [ ! -b /dev/md/ephemeral ]; then
echo "info: creating md dev"
sudo mdadm --create /dev/md/ephemeral \
--force \
--name=ephemeral \
--level=0 \
--raid-devices=$num_nvme \
$nvme_devices
else
echo "info: md dev already exists"
fi
sudo udevadm settle
# Check if the RAID device is already formatted
if ! sudo blkid -p -u filesystem /dev/md/ephemeral > /dev/null 2>&1; then
echo "info: creating xfs fs on md dev"
sudo mkfs.xfs /dev/md/ephemeral
else
echo "info: md dev is already formatted with an xfs fs"
fi
# Create mount point if it doesn't exist
if [ ! -d /raid0 ]; then
echo "info: creating mount point /raid0"
sudo mkdir /raid0
fi
# Mount the device if it's not already mounted
if ! mountpoint -q /raid0; then
echo "info: mounting /dev/md/ephemeral at /raid0"
sudo mount /dev/md/ephemeral /raid0
else
echo "info: /raid0 is already a mount point"
fi
echo "info: setup complete. Filesystem is mounted at /raid0."
```
---
# Managing Persistent Disks
## Creating Persistent Disks
**CLI:**
Use the `storage disks create` command to create a disk with your size. In the example below we will create 100 GiB disk called "data-1."
```
crusoe storage disks create \
--name data-1 \
--size 100GiB \
--location us-southcentral1-a \
--block-size 4096
```
`name`, `size`, and `location` are required arguments. The `block-size` can be either 512B or 4096B (default). When attaching a disk to a VM, the disk must be in the same location as the VM.
**UI:**
To create a disk via the [console](https://console.crusoecloud.com):
1. From the console, select **Storage** > **[Disks](https://console.crusoecloud.com/storage/disks)** in the left nav.
2. Click **Create Disk**.
3. Input a name for the disk, using only letters, numbers, `-` and `_`.
4. Set the desired size of the disk from 1GiB to 10TiB.
5. Set the desired block size of the disk—either 512B or 4096B.
6. Click **Create**.
**Terraform:**
Although Crusoe VMs come enabled with a 128 GB OS disk that is persistent, this disk is often not large enough for most AI or ML applications. For this reason, creating and attaching a persistent Disk is fundamental to Crusoe Cloud. The following is intended to help get you started in using Terraform to provision a persistent disk and attach the disk to a VM in Crusoe Cloud.
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates a Disk:
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_storage_disk" "new_data_disk" {
name = "new-data-disk"
size = "200GiB" // "1GiB" to "10TiB"
location = "us-southcentral1-a"
block_size = 4096 // or 512
}
```
`name`, `size`, and `location` are required arguments.
`name` can only include lowercase ascii characters, numbers and `-`.
`size` must be in format [Number][unit] where valid units are GiB (gibibyte) and TiB (tebibyte). Acceptable sizes are from 1GiB to 10TiB (required).
`location` of an attached disk must be the same as its attached VM.
`block_size` can be either 512B or 4096B (default). We recommend 512 for OS disks and 4096 for data disks.
## Viewing all disks
**CLI:**
Use the `storage disks list` command to list existing disks.
```sh
crusoe storage disks list
```
**UI:**
To view a list of existing disks via the [console](https://console.crusoecloud.com), select **Storage** > **[Disks](https://console.crusoecloud.com/storage/disks)** in the left nav.
**Terraform:**
To list existing disks using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.
```
# list disks
data "crusoe_storage_disks" "disks" {}
output "crusoe_disks" {
value = data.crusoe_storage_disks.disks
}
```
## Update an existing disk
:::info
Resizing a persistent disk requires the disk to be detached from the VM and the VM to be shutdown. Shrinking a persistent disk is not supported.
:::
**CLI:**
Use the `storage disks resize ` command to resize existing disks using the `--size` flag. Here's an example:
```sh
crusoe storage disks resize --size
```
**UI:**
To update a disk via the [console](https://console.crusoecloud.com):
1. From the console, select **Storage** > **[Disks](https://console.crusoecloud.com/storage/disks)** in the left nav.
2. Navigate to the row of the disk you want to update.
3. Click the plus (+) icon on the far right side of the row.
4. Enter the new size of the disk (disk size can only be increased).
5. Click **Confirm**.
**Terraform:**
To update an existing disk using the Crusoe Terraform provider, you can change the fields of an existing disk resource and run `terraform apply`. The Crusoe Terraform provider will apply the changes to the disk.
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_storage_disk" "new_data_disk" {
name = "new-data-disk"
size = "200GiB" -> "400GiB"
location = "us-southcentral1-a"
}
```
Currently, only the "size" of the disk can be changed. Changes to the "name" or "location" of the disk will force a re-creation of the disk (deletion and then creation of a new disk).
## Growing the filesystem after a resize
Resizing a Persistent Disk changes the size of the underlying block device, but does not automatically grow the filesystem inside the VM. After the disk is re-attached (and the VM is started, if it was stopped), extend the filesystem to match the new block device size.
For ext4 filesystems:
```sh
sudo resize2fs /dev/vdX
```
For xfs filesystems (the filesystem must be mounted):
```sh
sudo xfs_growfs /path/to/mount
```
Replace `/dev/vdX` or `/path/to/mount` with the actual device or mount point for the resized disk. Use `lsblk` and `df -h` to confirm the filesystem now reflects the new size.
## Deleting a disk
:::warning
Deleting a disk is a permanent action. The disk needs to be detached before it can be deleted.
:::
**CLI:**
Use the `storage disks delete ` command to delete a disk of your choice. As an example, you can delete a disk by replacing `DISK_NAME` with the name of the disk you wish to delete:
```
crusoe storage disks delete DISK_NAME
```
**UI:**
To delete a disk via the [console](https://console.crusoecloud.com):
1. From the console, select **Storage** > **[Disks](https://console.crusoecloud.com/storage/disks)** in the left nav.
2. Navigate to the row of the disk you wish to delete.
3. Click the trash can icon on the far right side of the row.
4. Enter the name of the disk you wish to delete in the popup that appears.
5. Click **Confirm**.
**Terraform:**
A disk can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
## Attaching Persistent Disks
**CLI:**
Use the `compute vms attach-disks` command to attach a disk to an instance. You can attach multiple disks to an instance with this command as well using a comma separated list of disk names.
```
crusoe compute vms attach-disks my-vm --disk name=data-1,mode=read-write
```
Use the `compute vms detach-disks` command to detach a disk from an instance. You can detach multiple disks from an instance with this command as well using a comma separated list of disk names.
```
crusoe compute vms detach-disks my-vm --disk name=data-1
```
**UI:**
To attach a disk to an instance via the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Click on the instance that you want to attach the disk to and view its details.
3. Under the **Disks** section, find the **Attach** dropdown menu and select the disk that you want to attach.
4. Click **+** next to the dropdown menu.
5. Start the instance.
**Terraform:**
Now if you want to create a VM with a disk attached, you can copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates a VM that uses a single Nvidia L40S GPU called `my-new-vm` with the disk `new-data-disk` created and attached in the `us-southcentral1-a` location:
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
locals {
my_ssh_key = file("~/.ssh/id_ed25519.pub")
}
resource "crusoe_storage_disk" "new_data_disk" {
name = "new-data-disk"
size = "200GiB"
location = "us-southcentral1-a"
}
// new VM
resource "crusoe_compute_instance" "my_vm" {
name = "my-new-vm"
type = "l40s-48gb.1x"
location = "us-southcentral1-a"
# specify the base image
image = "ubuntu22.04:latest"
disks = [
// disk attached at startup
{
id = crusoe_storage_disk.new_data_disk.id
mode = "read-write" // other option: "read-only"
attachment_type = "data"
}
]
ssh_key = local.my_ssh_key
}
```
In the `disks` section of the `VM` resource, `id`, `mode` and `attachment_type` are required.
`id` is the id of the disk, which you can append as `crusoe_storage_disk.new_data_disk.id`.
`mode` is either `read-only` or `read-write`.
`attachment_type` is only set to `data` currently.
After saving the code to a main.tf file, the following commands serve as the process to create a resource in Crusoe Cloud using Terraform:
`terraform init` - Initializes a working directory containing Terraform configuration files.
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will create the resources.
You can confirm that terraform successfully created the resources through the console, but if you prefer CLI, you can also run:
`crusoe storage disks list`
Which will show you the Disks you have created in your account.
## Formatting and Mounting Persistent Disks
Once the instance is started, login and use the command `lsblk` to inspect the available disks attached to the instance. These persistent ssd disks will have the `vd[b-z]` name. To associate the right disk name in the console corresponding to the persistent disk in the VM, you can navigate to `/dev/disk/by-id` and match `virtio-...` with the serial number for that disk in the console under the disk details section of the instance details page. For example:
```sh
ubuntu@:~$ ls -al /dev/disk/by-id/virtio-39734E8567D2ECA55C1
lrwxrwxrwx 1 root root 9 Jun 14 17:52 virtio-39734E8567D2ECA55C1 -> ../../vdb
```
`39734E8567D2ECA55C1` matches the serial number in the Crusoe Cloud Instance details console.
Create the filesystem on the block device by running:
```sh
sudo mkfs.ext4 /dev/vdb
```
Mount the volume by creating a `/scratch` directory and mounting `/dev/vdb` to `/scratch`
```sh
sudo mount -t ext4 /dev/vdb /scratch
```
Some additional `mount` options exist within the ext4 filesystem by passing `-O ...` for example `-O noatime,nodiratime,data=writeback` will avoid writing access times, as well disabling journaling if your workloads can benefit from these optimizations.
To mount Persistent Disks persistently across VM reboots, add an entry to the /etc/fstab file. Using the disk's UUID is recommended over the device path (e.g., `/dev/vdb`), since device names can change between reboots. Get the UUID with `blkid`:
```sh
ubuntu@:~$ sudo blkid /dev/vdb
/dev/vdb: UUID="abcd1234-ef56-7890-abcd-1234567890ab" TYPE="ext4"
```
:::warning
Always take a backup of the fstab file for recovery purposes and ensure [serial console access](../../compute/virtual-machines/accessing-vms.md#serial-console-access) is enabled to recover the VM in case of boot failures due to incorrect fstab entries.
:::
```sh
ubuntu@:~$ sudo vi /etc/fstab
...
UUID=abcd1234-ef56-7890-abcd-1234567890ab /scratch ext4 defaults,nofail,x-systemd.device-timeout=30 0 2
...
```
Verify the fstab entry mounts correctly
```sh
ubuntu@:~$ sudo mount -va
...
/scratch : successfully mounted
...
```
## Detaching Persistent Disks
When detaching a disk from an instance where the VM is still running, **it is critical to unmount the disk from the VM to avoid potential data loss**. Unmount the volume by using the `umount` command, such as follows:
```sh
sudo umount /dev/vdb
```
If the disk is detached from the instance before a successful `umount` command is run, the disk is at risk of data loss. In some rare cases, **the VM may exhibit unexpected behavior, such as a VM crash leading to loss of ephemeral disk data as well**.
## API reference
To manage Persistent Disks programmatically over HTTP, see the [Crusoe Cloud API reference](../../reference/api/index.md).
---
# Managing Shared Disks
## Creating Shared Disks
**CLI:**
Use the `storage disks create` command and specify the type "shared-volume" to create a Shared Disk of your specified size. In the example below we will create 1 TiB disk called "shared-1."
```
crusoe storage disks create \
--name shared-1 \
--type shared-volume \
--size 1TiB \
--location us-southcentral1-a
```
`name`, `size`, `type` and `location` are required arguments to create a Shared Disk. When attaching a disk to a VM, the disk must be in the same location as the VM.
**UI:**
To create a shared disk via the [console](https://console.crusoecloud.com):
1. From the console, select **Storage** > **[Disks](https://console.crusoecloud.com/storage/disks)** in the left nav.
2. Click **Create Disk**.
3. Input a name for the disk, using only letters, numbers, `-` and `_`.
4. Select the type **Shared Volume**.
5. Set the desired size of the disk from 1TiB to 1000TiB.
6. Click **Create**.
**Terraform:**
The following is intended to help get you started in using Terraform to provision a Shared disk and attach the disk to a VM in Crusoe Cloud.
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates a Disk:
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_storage_disk" "new_shared_disk" {
name = "new-shared-disk"
size = "1TiB" // "1TiB" to "1000TiB"
location = "us-southcentral1-a"
type = "shared-volume"
}
```
`name`, `size`, `type` and `location` are required arguments.
`name` can only include lowercase ascii characters, numbers and `-`.
`size` must be in format [Number][unit] where valid units are TiB (tebibyte). Acceptable sizes are from 1TiB to 1000TiB (required).
When attaching a disk to a VM, the disk must be in the same location as the VM.
## Viewing all Shared Disks
**CLI:**
Use the `storage disks list` command to list existing disks.
```sh
crusoe storage disks list
```
**UI:**
To view a list of existing disks via the [console](https://console.crusoecloud.com), select **Storage** > **[Disks](https://console.crusoecloud.com/storage/disks)** in the left nav.
**Terraform:**
To list existing disks using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.
```
# list disks
data "crusoe_storage_disks" "disks" {}
output "crusoe_disks" {
value = data.crusoe_storage_disks.disks
}
```
## Update an existing Shared Disk
**CLI:**
Use the `storage disks resize ` command to resize an existing Shared Disk using the `--size` flag. Specify `` in TiB increments (e.g., `5TiB`).
```
crusoe storage disks resize --size
```
**UI:**
To update a shared disk via the [console](https://console.crusoecloud.com):
1. From the console, select **Storage** > **[Disks](https://console.crusoecloud.com/storage/disks)** in the left nav.
2. Navigate to the row of the disk you wish to update.
3. Click the plus (+) icon on the far right side of the row.
4. Enter the new size of the disk.
5. Click **Confirm**.
**Terraform:**
To update an existing Shared Disk using the Crusoe Terraform provider, you can change the fields of an existing disk resource and run `terraform apply`. The Crusoe Terraform provider will apply the changes to the disk.
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_storage_disk" "new_shared_disk" {
name = "new-shared-disk"
size = "1TiB" -> "2TiB"
location = "us-southcentral1-a"
}
```
Currently, only the "size" of the disk can be changed. Changes to the "name" or "location" of the disk will force a re-creation of the disk (deletion and then creation of a new disk).
Shared Disks' size can be increased or decreased in increments of 1 TiB with a minimum size of 1 TiB and a maximum size of 1000 TiB. Decreasing the size of a Shared Disk is only allowed up to the nearest rounded-up size in TiB. Any operation to reduce the size lower than the used capacity will fail.
## Deleting a Shared Disk
:::warning
Deleting a disk is a permanent action.
All Crusoe VMs must be unmounted from a Shared Disk before the Shared Disk can be deleted.
:::
**CLI:**
Use the `storage disks delete ` command to delete a disk of your choice. As an example, you can delete a disk by replacing `DISK_NAME` with the name of the disk you wish to delete:
```
crusoe storage disks delete DISK_NAME
```
**UI:**
To delete a disk via the [console](https://console.crusoecloud.com):
1. From the console, select **Storage** > **[Disks](https://console.crusoecloud.com/storage/disks)** in the left nav.
2. Navigate to the row of the disk you wish to delete.
3. Click the trash can icon on the far right side of the row.
4. Enter the name of the disk you wish to delete in the popup that appears.
5. Click **Confirm**.
**Terraform:**
A disk can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
## Mounting Shared Disks
Before mounting a Shared Disk on a VM for the first time, install the VAST NFS driver — see [Setting up the VAST NFS driver](./setup-nfs-driver.mdx).
Confirm the driver is installed on the VM:
```sh
ubuntu@:~$ vastnfs-ctl status
version: 4.0.35-vastdata
kernel modules: sunrpc
services: rpcbind.socket rpcbind
rpc_pipefs: /run/rpc_pipefs
```
The mount command for the disk can be found in the Crusoe Cloud Console, either on the compute instance details page under the disk actions section or on the storage page under the actions section.
Not all disks can be mounted using the same endpoint, so be mindful of the endpoint (DNS name or IP range) exposed for your disk.
```sh
# Example disk mount using DNS
ubuntu@:~$ sudo mount -t nfs -o vers=3,nconnect=16,spread_reads,spread_writes,remoteports=dns nfs.crusoecloudcompute.com:/volumes/
# Example disk mount using IPs
ubuntu@:~$ sudo mount -t nfs -o vers=3,nconnect=16,spread_reads,spread_writes,remoteports=172.27.255.2-172.27.255.17 172.27.255.2:/volumes/
```
:::info
The `remoteports=dns` form requires the VM to resolve the DNS in your disk's displayed mount command. If your VPC uses custom DNS, verify resolution first (for example, `dig nfs.crusoecloudcompute.com`) before relying on this mount command.
:::
The `findmnt` command is used to confirm the Shared Disk is mounted correctly
```sh
ubuntu@:~$ findmnt -t nfs
TARGET SOURCE FSTYPE OPTIONS
nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,forcerdirplus,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=172.27.255.2,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,spread_reads,spread_writes,addr=172.27.255.2
```
Running `df` shows correct provisioned capacity for the Shared Disk
```sh
ubuntu@:~$ df -h
Filesystem Size Used Avail Use% Mounted on
100T 52G 101T 1%
```
To mount Shared Disks persistently across VM reboots, add an entry to the /etc/fstab file
:::warning
Always take a backup of the fstab file for recovery purposes and ensure [serial console access](../../compute/virtual-machines/accessing-vms.md#serial-console-access) is enabled to recover the VM in case of boot failures due to incorrect fstab entries.
:::
```sh
ubuntu@:~$ sudo vi /etc/fstab
...
# eu-iceland1-a
nfs.crusoecloudcompute.com:/volumes/ nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,forcerdirplus,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,local_lock=none,remoteports=dns,spread_reads,spread_writes,_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30 0 0
# all other regions
172.27.255.2:/volumes/ nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,forcerdirplus,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,local_lock=none,spread_reads,spread_writes,_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30,remoteports=172.27.255.2-172.27.255.17 0 0
...
```
Verify if the automount works for Shared Disks
```sh
ubuntu@:~$ sudo mount -va
...
: successfully mounted
...
```
## Unmounting Shared Disks
Shared Disks can be unmounted by running umount command:
```sh
ubuntu@:~$ sudo umount
ubuntu@:~$ findmnt -t nfs
ubuntu@:~$
```
## Benchmarking a mounted Shared Disk
Ensure that the readahead parameter is set and MTU is set to 9000 before running the fio benchmark, for best results.
The fio tool can be used to benchmark Shared Disks. From within a mounted Shared Disk, you can run the following commands to test read/write bandwidth and IOPS:
```sh
# test write bw
fio --name=my-job --group_reporting --time_based=1 --cpus_allowed_policy=split --runtime=10s --ramp_time=5s --size 20G --numjobs=32 --ioengine=aio --direct=1 --iodepth 8 --rw write --bs 1m
# test read bw
fio --name=my-job --group_reporting --time_based=1 --cpus_allowed_policy=split --runtime=10s --ramp_time=5s --size 20G --numjobs=32 --ioengine=aio --direct=1 --iodepth 8 --rw read --bs 1m
# test write iop
fio --name=my-job --group_reporting --time_based=1 --cpus_allowed_policy=split --runtime=10s --ramp_time=5s --size 20G --numjobs=32 --ioengine=aio --direct=1 --iodepth 8 --rw write --bs 4k
# test read iop
fio --name=my-job --group_reporting --time_based=1 --cpus_allowed_policy=split --runtime=10s --ramp_time=5s --size 20G --numjobs=32 --ioengine=aio --direct=1 --iodepth 8 --rw read --bs 4k
```
## Shared Disks Performance Profile
Shared Disks performance is designed to scale with capacity. The performance target of the Shared Disk is determined by its capacity, scaling linearly up to 1 PiB from 100 TiB. This table shows the scaling rate per 1 TiB of provisioned capacity.
| **Metric** | **Read Bandwidth** | **Write Bandwidth** | **IOPS** |
| :----------- | :--------------------- | :-------------------- | :---------------------- |
| Scaling Rate | Up to 200 MB/s per TiB | Up to 40 MB/s per TiB | Up to 1.2k IOPs per TiB |
While performance scales linearly per TiB, the service includes a base level of performance (at the 100 TiB level) and a defined maximum ceiling (at the 1 PiB level). Disks smaller than 100 TiB will still receive the base performance.
| **Metric** | **Base Performance (At 100 TiB)** | **Maximum Performance (At 1 PiB)** |
| :------------------ | :-------------------------------- | :--------------------------------- |
| **Read Bandwidth** | Up to 20 GB/s | Up to 200 GB/s |
| **Write Bandwidth** | Up to 4 GB/s | Up to 40 GB/s |
| **IOPS** | Up to 120,000 IOPs | Up to 1,200,000 IOPs |
### Aggregate Performance vs. Per-VM Performance
The performance metrics (Read, Write, and IOPS) detailed above represent the target aggregate performance of the shared disk across all virtual machines (VMs) attached to it. Performance within individual VMs will vary based on the available [VPC Network Bandwidth](../../compute/virtual-machines/overview.md) allocated to that specific VM .
## API reference
To manage Shared Disks programmatically over HTTP, see the [Crusoe Cloud API reference](../../reference/api/index.md).
---
# Setting up the VAST NFS driver
Shared Disks are mounted over NFS. Before mounting a Shared Disk on a VM for the first time, verify that the VAST NFS driver is installed on that VM. If your VM was created after Dec 2025, the driver is likely already installed in the VM image.
## Verify the driver is installed
Run `vastnfs-ctl status` on the VM:
```sh
ubuntu@:~$ vastnfs-ctl status
version: 4.0.35-vastdata
kernel modules: sunrpc
services: rpcbind.socket rpcbind
rpc_pipefs: /run/rpc_pipefs
```
If the command fails with `command not found`, follow the install steps below.
## Install the driver on a single VM
Use the Python installer published by Crusoe:
```sh
wget -O crusoe_shared_disks_nfs_setup.py https://github.com/crusoecloud/crusoe-nfs-support/raw/refs/heads/main/setup/crusoe_shared_disks_nfs_setup.py
python3 crusoe_shared_disks_nfs_setup.py --apply-network-optimizations --apply-read-ahead-cache
```
:::info
`--apply-network-optimizations` sets the MTU to 9000 and applies ring-buffer optimizations. `--apply-read-ahead-cache` increases the NFS readahead cache from the Linux default of 128KB to 16MB. Both flags are recommended for best performance.
:::
## Install on multiple VMs with pssh
To apply the installer across many VMs at once, use `pssh`.
Install `pssh` on a Mac:
```sh
brew install pssh
```
Install `pssh` on Linux:
```sh
sudo apt install pssh
```
Create a file named `hosts.txt` listing the VMs, one per line:
```
ubuntu@
ubuntu@
```
:::info
Ensure SSH connectivity is set up between your workstation and each VM before running the commands below.
:::
On a Mac (uses `pscp` / `pssh`):
```sh
pscp -h hosts.txt crusoe_shared_disks_nfs_setup.py /home/ubuntu/crusoe_shared_disks_nfs_setup.py
pssh -t 0 -h hosts.txt "export DEBIAN_FRONTEND=noninteractive && python3 /home/ubuntu/crusoe_shared_disks_nfs_setup.py --apply-network-optimizations --apply-read-ahead-cache -y"
```
On Linux (uses `parallel-scp` / `parallel-ssh`):
```sh
parallel-scp -h hosts.txt crusoe_shared_disks_nfs_setup.py /home/ubuntu/crusoe_shared_disks_nfs_setup.py
parallel-ssh -t 0 -h hosts.txt "export DEBIAN_FRONTEND=noninteractive && python3 /home/ubuntu/crusoe_shared_disks_nfs_setup.py --apply-network-optimizations --apply-read-ahead-cache -y"
```
Once the driver is installed on all VMs, proceed to [Mounting Shared Disks](./managing-shared-disks.mdx#mounting-shared-disks).
---
# Shared Disks Metrics
Shared Disk metrics provide visibility into the IOPS, bandwidth, latency, and capacity utilization of your shared disks. Unlike VM and CMK metrics, shared disk metrics are available out-of-the-box with no Crusoe Watch Agent installation required. Metrics are collected every 5 minutes and retained for 30 days.
Shared Disk metrics are accessible via the Console, API, Telemetry Conduit, and Crusoe MCP.
**Console:** Navigate to **Storage** in the left navigation bar and select the shared disk you want to inspect.
**API, Grafana, Telemetry Conduit, and Crusoe MCP:** See [Get started](../../command-center/get-started.mdx) for instructions on generating a monitoring token and using each access method.
## Available Metrics
| **Metric** | **Query Parameter** | **Suggested PromQL Query** |
| --------------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------- |
| Disk read IOPS | crusoe_sdisk_disk_read_iops | `rate(crusoe_sdisk_disk_read_iops_count[5m])` |
| Disk write IOPS | crusoe_sdisk_disk_write_iops | `rate(crusoe_sdisk_disk_write_iops_count[5m])` |
| Disk read bandwidth (bytes per second) | crusoe_sdisk_disk_read_bw_bytes_sum | `rate(crusoe_sdisk_disk_read_bw_bytes_sum[5m])` |
| Disk write bandwidth (bytes per second) | crusoe_sdisk_disk_write_bw_bytes_sum | `rate(crusoe_sdisk_disk_write_bw_bytes_sum[5m])` |
| Disk read latency | crusoe_sdisk_disk_read_latency | `rate(crusoe_sdisk_disk_read_latency_sum[5m]) / rate(crusoe_sdisk_disk_read_latency_count[5m])` |
| Disk write latency | crusoe_sdisk_disk_write_latency | `rate(crusoe_sdisk_disk_write_latency_sum[5m]) / rate(crusoe_sdisk_disk_write_latency_count[5m])` |
| Used disk capacity (bytes) | crusoe_sdisk_disk_capacity_used_byte | `crusoe_sdisk_disk_capacity_used_byte` |
By default, queries return metrics for all disks within a project. To query metrics for a specific disk, add a label selector:
```
rate(crusoe_sdisk_disk_write_iops_count{disk_id="your-disk-id"}[5m])
```
For token generation and querying metrics via API, Grafana, or Telemetry Conduit, see [Get started](../../command-center/get-started.mdx).
---
# Troubleshooting
Common issues you may encounter when working with Ephemeral, Persistent, or Shared Disks and how to resolve them.
---
## Resized a Persistent Disk but the filesystem did not grow
**Cause**: Resizing a Persistent Disk changes the underlying block device size, but does not grow the filesystem inside the VM. The filesystem must be extended manually after the disk is re-attached.
**Solution**: Follow the steps in [Growing the filesystem after a resize](./managing-persistent-disks.mdx#growing-the-filesystem-after-a-resize).
---
## `vastnfs-ctl: command not found`
**Cause**: The VAST NFS driver required for mounting Shared Disks is not installed on the VM.
**Solution**: Install the driver using the steps in [Setting up the VAST NFS driver](./setup-nfs-driver.mdx).
---
## `umount: target is busy` when unmounting a Shared Disk
**Cause**: A process on the VM has an open file handle or working directory on the mount path, so the kernel cannot release the mount.
**Solution**: Find and stop the processes holding the mount:
```sh
sudo lsof +f -- /path/to/mount
sudo fuser -m /path/to/mount
```
Then retry `sudo umount /path/to/mount`. If the processes cannot be stopped, `sudo umount -l /path/to/mount` performs a lazy unmount that completes when all handles are closed.
---
## VM fails to boot after editing `/etc/fstab`
**Cause**: A malformed or unresolvable entry in `/etc/fstab` blocks the boot process while waiting for the mount.
**Solution**: Use [serial console access](../../compute/virtual-machines/accessing-vms.md#serial-console-access) to drop into single-user mode and restore `/etc/fstab` from a backup, or comment out the offending line. To reduce the risk of boot failures, include the `nofail` and `x-systemd.automount` options on Shared Disk entries — see the [fstab example](./managing-shared-disks.mdx#mounting-shared-disks).
---
## Persistent Disk does not appear in `lsblk` after attaching
**Cause**: The VM kernel has not yet picked up the newly attached device, or the attach operation is still in progress on the backend.
**Solution**:
1. Confirm the disk is attached in the Crusoe Cloud console or via `crusoe storage disks list`.
2. Run `lsblk` again after a few seconds.
3. If the disk is still missing, rescan the virtio bus:
```sh
echo 1 | sudo tee /sys/bus/pci/rescan
```
4. If the disk remains invisible, detach and re-attach the disk via the console, CLI, or Terraform.
---
## VM crashed after detaching a Persistent Disk
**Cause**: The disk was detached while still mounted. Detaching a mounted Persistent Disk can cause the VM kernel to panic and, in some cases, results in loss of data on both the persistent and ephemeral disks.
**Solution**: Always unmount the disk before detaching:
```sh
sudo umount /dev/vdX
```
See [Detaching Persistent Disks](./managing-persistent-disks.mdx#detaching-persistent-disks).
---
# VPC Network overview
VPC Networks and VPC Subnets form the basis of VM-to-VM communication on Crusoe Cloud. They represent an ethernet network, which is a single, non-blocking fabric across all VMs in a given location.
## Firewall Rules overview
Firewall Rules allow you to control network access to your VMs, both from the public internet as well as from other Crusoe Cloud VMs in the same network. To learn more about default Firewall Rules and creating your own rules within your organization, please visit [this page](../networking/firewall-rules/overview.md).
## InfiniBand Network overview
InfiniBand networks form the basis of high-performance GPU-to-GPU communication on Crusoe Cloud.
InfiniBand Networks are defined by a set of physical machines on the same physical InfiniBand fabric (in the same physical location). Developers are therefore unable to create "new" InfiniBand networks, and can instead create partitions, which are the security barrier between machines on the same network. There is no equivalent to Firewall Rules on an InfiniBand network.
## Networking billing
At this time, Crusoe Cloud does not charge for network ingress or egress, either within a VPC or to/from the public internet. InfiniBand networking, where applicable, is included in the cost of any InfiniBand enabled VMs.
---
# Overview
# VPC Networks Overview
Crusoe Cloud provides a high performance Software Defined Network (SDN) for developers.
## Concepts
### Default networks and subnets
When you create a project, Crusoe creates a default VPC network and default zonal VPC subnets for all regions to which you have access. The CIDRs for these are:
| Network | CIDR |
| --------------------- | --------------- |
| `default-vpc-network` | `172.27.0.0/16` |
| Subnets | CIDR |
| ----------------------------------- | ---------------- |
| `default-subnet-us-east1-a` | `172.27.16.0/20` |
| `default-subnet-us-southcentral1-a` | `172.27.32.0/20` |
| `default-subnet-eu-iceland1-a` | `172.27.48.0/20` |
| `default-subnet-us-west1-a` | `172.27.64.0/20` |
:::info
Certain regions listed above are restricted and may not be available for immediate provisioning. If you require access to additional locations, please [contact our sales team](https://crusoe.ai/contact-us#sales) to discuss your use case.
:::
Crusoe also creates [default firewall rules](../firewall-rules/overview.md#default-firewall-rules) for this network and the associated subnets. Crusoe manages adding new subnets as new zones and regions come online, and will also add new firewall rules to the default network to cover added subnets.
### Non-default networks and subnets
If you need more control over your infrastructure, you can also create non-default networks and subnets. Non-default subnets can be created with IP ranges in [RFC 1918](https://datatracker.ietf.org/doc/html/rfc1918) space or in DoD-assigned `/8` blocks:
| IP range type | CIDRs |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| RFC 1918 (recommended) | `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` |
| DoD blocks | `6.0.0.0/8`, `7.0.0.0/8`, `11.0.0.0/8`, `21.0.0.0/8`, `22.0.0.0/8`, `26.0.0.0/8`, `28.0.0.0/8`, `29.0.0.0/8`, `30.0.0.0/8`, `33.0.0.0/8`, `55.0.0.0/8`, `214.0.0.0/8`, `215.0.0.0/8` |
:::warning
DoD blocks are intended for organizations that control allocations in those ranges, not as a workaround for exhausted RFC 1918 space.
:::
Non-default networks and subnets do not come with any default firewall rules, so you must add all applicable firewall rules to the network.
### NAT Gateways
:::info
This capability is currently in Limited Availability, and may not be visible to your organization by default. To request access, please [contact support](../../resources/support.md).
:::
NAT Gateways provide a path to egress for Crusoe Cloud instances that are not assigned a public IP address. This is typically employed when you want to limit access from the public internet to your instances, but want to allow those instances to download data such as model artifacts or images.
NAT Gateways can be enabled on a subnet basis. When enabled, all instances that do not have a public IP assigned within that subnet will be automatically configured to route any outbound traffic to the internet via the NAT Gateway.
### Reserved IPs within subnets
We currently reserve the first five IPs within a subnet (e.g. `172.27.0.0` through `172.27.0.4`) as well as the broadcast address (e.g. `172.27.0.255`). This applies to all subnets, both the default subnets as well as any non-default subnets.
### Internal DNS
By default, we provide internal DNS for all VMs within a VPC network. VMs are reachable at `$VM_NAME.$LOCATION.compute.internal`, e.g. `stable-diffusion-serving.us-southcentral1-a.compute.internal`. Note that internal DNS is provided on a per-network basis.
## Limitations
### IPv4 only
Currently, we only support IPv4.
### Static IPs
Currently, all private IPs are static. All public IPs are dynamic by default, which means they will change during a VM stop and restart, but can be changed to static. Learn more on [how to updated the public IP type](../../compute/virtual-machines/managing-vms.mdx).
### Communication limited to within a region
Currently, subnet-to-subnet communication using private IP addresses is limited to communication between instances and subnets in the same region. For instance, two VMs in the `default-subnet-us-east1-a` are allowed to communicate (provided that the correct firewall rules are configured to allow this communication), as are `instance-1` in a non-default `my-subnet-1-us-east1-a` and `instance-2` in a non-default `my-subnet-2-us-east1-a`. Communication between `instance-1` in `default-subnet-us-east1-a` and `instance-2` in `default-subnet-us-southcentral1-a`, over their private IPs, is not allowed.
If you want instances in different regions to talk to each other, you can use the public IP addresses assigned to the instances. We suggest configuring firewall rules to allow for secure connections across regions.
---
# Managing VPC Networks
# Managing your networks and subnets
## Creating a new VPC network
**CLI:**
Use the `networking vpc-networks create` command to create a new VPC network.
```sh
crusoe networking vpc-networks create \
--name my-new-vpc-network \
--cidr 10.0.0.0/8
```
**UI:**
To create a VPC network via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[VPC Networks](https://console.crusoecloud.com/networking/networks)** in the left nav.
2. Click **Create VPC Network**.
3. Input a name and CIDR range for the VPC Network.
4. Click **Create**.
**Terraform:**
To create a VPC network using the Crusoe Terraform provider, you can use the following code snippet:
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_vpc_network" "my_vpc_network" {
name = "my-new-network"
cidr = "10.0.0.0/8"
}
```
"name" and "cidr" required arguments for the VPC network resource in the Crusoe Terraform provider.
## Viewing all VPC networks
**CLI:**
Use the `networking vpc-networks list` command to list existing networks.
```sh
crusoe networking vpc-networks list
```
**UI:**
To view networks via the [console](https://console.crusoecloud.com), select **Networking** > **[VPC Networks](https://console.crusoecloud.com/networking/networks)** in the left nav.
**Terraform:**
To list existing VPC networks using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.
```hcl
# list vpc networks
data "crusoe_vpc_networks" "networks" {}
output "crusoe_vpc_networks" {
value = data.crusoe_vpc_networks.networks
}
```
## Update an existing VPC network
**CLI:**
Non-default VPC networks can be updated in the CLI using the `networking vpc-networks update ` command. Currently, only the name of the non-default VPC network can be updated (using the `--name` flag).
**UI:**
To update an existing non-default VPC network via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[VPC Networks](https://console.crusoecloud.com/networking/networks)** in the left nav.
2. Navigate to the row of the VPC network you want to update.
3. Click the pencil icon on the far right side of the row.
4. Edit the fields you want to modify.
5. Click **Update** to save your changes.
**Terraform:**
To update a VPC network using the Crusoe Terraform provider, you can change the fields of an existing VPC network resource and run `terraform apply`. The Crusoe Terraform provider will apply the changes to the VPC network.
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_vpc_network" "my_vpc_network" {
name = "my-new-network" -> "new-network-name"
cidr = "10.0.0.0/8"
}
```
Currently, only the "name" of the VPC network can be updated. Changes to the "cidr" of the VPC network will force a re-creation of the VPC network (deletion and then creation of a new network)
## Deleting a VPC network
:::info
**Warning:** deleting a VPC network is a permanant action that will require re-creation of the network to recover.
:::
**CLI:**
Non-default VPC Networks can be deleted in the CLI using the `networking vpc-networks delete ` command.
**UI:**
To delete a VPC network via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[VPC Networks](https://console.crusoecloud.com/networking/networks)** in the left nav.
2. Navigate to the row of the VPC Network you want to delete.
3. Click the trash can icon on the far right side of the row.
4. Click **Confirm**.
**Terraform:**
A VPC network can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
If you are having issues working with your VPC networks, please [contact support](../../resources/support.md).
---
# Managing VPC Subnets
# Manage your subnets
## Creating a new VPC subnet
**CLI:**
Use the `networking vpc-subnets create` command to create a new VPC subnet. Optionally, you may use the `nat-gateway-enabled` flag to enable NAT Gateways.
```sh
crusoe networking vpc-subnets create \
--name my-new-vpc-subnet \
--location us-east1-a
--vpc-network-id
--cidr 10.10.0.0/16
--nat-gateway-enabled true
```
**UI:**
To create a VPC subnet via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[VPC Subnets](https://console.crusoecloud.com/networking/subnets)** in the left nav.
2. Click **Create VPC Subnet**.
3. Input a name and CIDR range for the VPC Subnet.
4. Select the location and VPC Network in which you would like to create the VPC Subnet.
5. (Optional) Use the **Enable NAT Gateway** toggle to provision your VPC Subnet with a NAT Gateway.
6. Click **Create**.
**Terraform:**
To create a VPC subnet using the Crusoe Terraform provider, you can use the following code snippet. The snippet creates a new VPC network, and then creates a new VPC subnet in the created. This example also creates a subnet with a NAT Gateway.
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_vpc_network" "my_vpc_network" {
name = "my-new-network"
cidr = "10.0.0.0/8"
}
resource "crusoe_vpc_subnet" "my_vpc_subnet" {
name = "my-new-subnet"
cidr = "10.0.0.0/16"
location = "us-northcentral1-a"
network = crusoe_vpc_network.my_vpc_network.id
nat_gateway_enabled = true
}
```
"name", "cidr", "location", and "network" are required arguments for the VPC subnet resource in the Crusoe Terraform provider.
## Viewing all VPC subnets
**CLI:**
Use the `networking vpc-subnets list` commands to list existing subnets.
```sh
crusoe networking vpc-subnets list
```
**UI:**
To view subnets via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[VPC Subnets](https://console.crusoecloud.com/networking/subnets)** in the left nav.
2. Click on a subnet to view additional details.
**Terraform:**
To list existing VPC subnets using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.
```hcl
# list vpc subnets
data "crusoe_vpc_subnets" "subnets" {}
output "crusoe_vpc_subnets" {
value = data.crusoe_vpc_subnets.subnets
}
```
## Update an existing VPC subnet
**CLI:**
Non-default VPC subnets can be updated in the CLI using the `networking vpc-subnets update ` command. You may update the name, or enable / disable NAT Gateways on the subnet.
```sh
crusoe networking vpc-subnets update \
--name my-new-vpc-subnet \
--nat-gateway-enabled true/false
```
**UI:**
To update an existing non-default VPC subnet via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[VPC Subnets](https://console.crusoecloud.com/networking/subnets)** in the left nav.
2. Navigate to the row of the VPC subnet you want to update.
3. Click the pencil icon on the far right side of the row.
4. Edit the fields you want to modify.
5. Click **Update** to save your changes.
**Terraform:**
To update an existing VPC subnet using the Crusoe Terraform provider, you can change the fields of an existing VPC subnet resource and run `terraform apply`. The Crusoe Terraform provider will apply the changes to the VPC subnet.
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
...
resource "crusoe_vpc_subnet" "my_vpc_subnet" {
name = "my-new-subnet" -> "my-new-name"
cidr = "10.0.0.0/16"
location = "us-northcentral1-a"
network = crusoe_vpc_network.my_vpc_network.id
}
```
Currently, only the "name" of the VPC subnet can be updated. Changes to the "cidr", "location", or "network" of the VPC subnet will force a re-creation of the VPC subnet (deletion and then creation of a new subnet)
## Deleting a VPC subnet
:::info
**Warning:** deleting a VPC subnet is a permanant action that will require re-creation of the subnet to recover.
:::
**CLI:**
VPC subnets can be deleted in the CLI using the `networking vpc-subnets delete ` command.
**UI:**
To delete a VPC subnet via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[VPC Subnets](https://console.crusoecloud.com/networking/subnets)** in the left nav.
2. Navigate to the row of the VPC Subnet you want to delete.
3. Click the trash can icon on the far right side of the row.
4. Click **Confirm**.
**Terraform:**
A VPC subnet can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
If you are having issues working with your VPC subnets, please
[contact support](../../resources/support.md).
---
# Overview
VPC Firewall Rules allow developers to granularly control access to VPC networks, subnets, and individual VMs.
## Concepts
VPC Firewall Rules rely on a five-tuple to filter L3 traffic and determine if it is allowed to pass:
- **Action:** what action to take for categories of traffic, `allow` (firewall rules implicitly deny traffic unless a rule explicitly allows the traffic)
- **Direction:** what direction traffic is heading, relative to the VM, `ingress` is "outside world to the VM" and `egress` is "VM to the outside world"
- **Protocols:** what protocols are filtered, `tcp`, `udp`, or `icmp`
- **Source:** the IP(s) and port(s) that traffic is "coming from" (the "outside world" in an `ingress` rule; the VM in an `egress` rule)
- **Destination:** the IP(s) and port(s) that traffic is "heading to" (the VM in an `ingress` rule or the "outside world" in an `egress` rule). Currently for ingress rules, you must specify the _private_ IP address of a destination VM (as opposed to the public IP).
By default, all traffic is denied unless a rule explicitly allows it.
## Default Firewall Rules
Crusoe provides the following default firewall rules in the default VPC network.
Ingress:
- `default-allow-ssh`: allow SSH access from the public internet to all instances
- `default-allow-icmp-internal`: allow ICMP traffic from all VMs on the same network; note this does not allow public ICMP traffic
- `default-allow-internal-network`: allow all TCP and UDP traffic from all VMs on the same network
Egress:
- `default-allow-icmp-egress`: allow all ICMP traffic from all VMs on the network to egress to the public internet
- `default-allow-tcp-udp-egress`: allow all TCP and UDP traffic from all VMs on the network to egress to the public internet
If you do not want to allow this traffic, you can delete one or all of these rules.
### Firewall Rules in non-default VPCs
Non-default (custom) VPC networks are not created with any default firewall rules. Firewall rules implicitly deny traffic unless a rule explicitly allows the traffic to pass, so all communication to/from non-default VPC network will be denied until firewall rules are added to allow desired traffic.
In order to allow ingress and egress communication for non-default VPC, explicit firewall rules have to be configured for ingress/egress. We recommend starting with the default firewall rules above, and modifying them as desired.
## Limitations
### Mixing protocols
At the current time, you can only create `tcp` and/or `udp`, or `icmp` rules. You cannot create a mix of `tcp` or `udp` with `icmp`. Similarly, you cannot add `ports` to an `icmp` rule.
### Allow only/implicit deny
At the current time, firewall rules only support `allow` rules. By default, all traffic is denied unless it is specifically allowed. We do not currently have plans to support `deny` rules.
### Port restrictions
For security and anti-spam reasons, by default, Crusoe does not permit outbound SMTP traffic on TCP ports 25, 465, or 2525.
For sending outbound email from machines, we recommend using SMTP Submission on TCP port 587 or using an email service provider that provides an API over HTTPS.
### Destination IPs for ingress rules
When allowing ingress traffic to VMs, the firewall rule's destination IP(s) must reference the _private_ IP(s) of a VM.
## Troubleshooting Firewall Rules
The following reminders may help if you are troubleshooting firewall rules:
- Source ports may not be the same as the destination ports (e.g. SSH may not come from port 22, but it will arrive at port 22)
---
# Managing firewall rules
## Manage your firewall rules
### Creating a new firewall rule
**CLI:**
Use the `networking firewall-rules create` command to create a firewall rule. As an example, you can create a firewall rule to allow HTTPS serving:
```sh
crusoe networking vpc-firewall-rules create \
--name allow-https \
--action ALLOW \
--destination-ports 443 \
--destinations 10.0.0.0/8 \
--protocols tcp,udp \
--source-ports 1-65535 \
--sources 0.0.0.0/0 \
--direction INGRESS \
--vpc-network-id NETWORK_ID
```
**UI:**
To create a firewall rule via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Firewall Rules](https://console.crusoecloud.com/networking/firewall-rules)** in the left nav.
2. Click **Create Firewall Rule**.
3. Add the required information.
4. Click **Create**.
**Terraform:**
Creating Firewall rules is fundamental to Crusoe Cloud for various reasons, including security and access control to and from your VM. The following is intended to help get you started in using Terraform to provision Firewall rules and attach the firewall rules to a VM in Crusoe Cloud.
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates a Firewall rule:
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_vpc_firewall_rule" "open_fw_rule" {
network = "9999999y-d1bb-4e19-9bdc-1fc38392f18x" // VPC network ID
name = "example-terraform-rule" // Name of Firewall rule
action = "allow"
direction = "ingress"
protocols = "tcp"
source = "0.0.0.0/0"
source_ports = "1-65535"
destination = "0.0.0.0/0"
destination_ports = "1-65535"
}
```
`network`, `name`, `action`, `direction`, `protocols`, `source`, `source_ports`, `destination`, `destination_ports` are required arguments.
`network` is the VPC network ID, which can be found by running crusoe networking vpc-networks list and copying and pasting the UUID.
`name` is the name of the firewall rule.
`action` is the action of the rule, right now “allow” is the only action.
`direction` is the direction you want traffic to go, right now “ingress” is the only direction.
`protocols` is what protocol is filtered, the options are tcp, udp, or icmp.
`source` is the IP (or IPs) that traffic is “coming from”.
`source_ports` are the Port (or Ports) that traffic is “coming from”.
`destination` is the IP (or IPs) that traffic is “heading to”. Use private IP address for destination VMs (as opposed to public IP).
`destination_ports` are the Port (or Ports) that traffic is “heading to”.
After saving the code to a `main.tf` file, the following commands serve as the process to create a resource in Crusoe Cloud using Terraform:
`terraform init` - Initializes a working directory containing Terraform configuration files.
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will create the resources.
## Viewing all existing firewall rules
**CLI:**
Use the `networking vpc-firewall-rules list` command to list all existing firewall rules.
```sh
crusoe networking vpc-firewall-rules list
```
**UI:**
To view firewall rules via the [console](https://console.crusoecloud.com), go to the [Firewall Rules](https://console.crusoecloud.com/networking/firewall-rules) page.
**Terraform:**
It is currently not possible to view firewall rules in the Crusoe Terraform provider.
### Update an existing firewall rule
**CLI:**
Use the `networking vpc-firewall-rules update RULE_ID` command to modify an existing firewall rule. Specify the resource ID of the rule you wish to update along with the fields to be modified.
```sh
crusoe networking vpc-firewall-rules update RULE_ID \
--name allow-https-v2 \
--destination-ports 443 \
--destinations 172.27.1.12 \
--protocols TCP,UDP \
--vpc-network-id NETWORK_ID
```
**UI:**
To update an existing firewall rule via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Firewall Rules](https://console.crusoecloud.com/networking/firewall-rules)** in the left nav.
2. Navigate to the row of the firewall rule you want to update.
3. Click the pencil icon on the far right side of the row.
4. Edit the fields you want to modify.
5. Click **Update** to save your changes.
**Terraform:**
```hcl
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
resource "crusoe_vpc_firewall_rule" "open_fw_rule" {
network = "9999999y-d1bb-4e19-9bdc-1fc38392f18x" // VPC network ID
name = "example-terraform-rule" // Name of Firewall rule
action = "allow"
direction = "ingress"
protocols = "tcp"
source = "0.0.0.0/0"
source_ports = "1-65535"
destination = "0.0.0.0/0"
destination_ports = "80" // only open HTTP port 80
}
```
After making any changes, save the code and then perform the following commands:
`terraform plan` - the output of this command will show the resources Terraform plans on creating.
`terraform apply` - this command will create the resources.
### Deleting a firewall rule
:::info
**Warning:** deleting a firewall rule is a permanant action that will require re-creation of the rule to recover.
:::
**CLI:**
Use the `networking firewall-rules delete` command to delete a specific firewall rule:
```sh
crusoe networking vpc-firewall-rules delete --name RULE_NAME
```
**UI:**
To delete a firewall rule via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Firewall Rules](https://console.crusoecloud.com/networking/firewall-rules)** in the left nav.
2. Navigate to the row of the firewall rule you want to delete.
3. Click the trash can icon on the far right side of the row.
4. Confirm deletion.
**Terraform:**
A firewall rule can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
If you are having issues creating or deleting firewall rules, please [contact support](../../resources/support.md).
---
# Managing InfiniBand networking
Crusoe Cloud supports high performance interconnects utilizing NVIDIA Mellanox InfiniBand (IB) networking. The fabric is currently supported for the instance types in the table below:
| Instance Type | Number of Infiniband HCAs per Instance | Total InfiniBand Bandwidth (Gbps) |
| ---------------------- | -------------------------------------- | --------------------------------- |
| `a100-80gb-sxm-ib.8x` | 8 | 1600 |
| `h100-80gb-sxm-ib.8x` | 8 | 3200 |
| `h200-141gb-sxm-ib.8x` | 8 | 3200 |
The general workflow, which will be discussed in more detail below, emcompasses selecting an IB network for the list of networks available within a location, and then creating an IB parition from within that network. Finally you will launch instances into that partition. This ensures cluster tenancy within a parition to maximize performance with cluster wide isolation.
## InfiniBand VM image
Crusoe Cloud provides a default VM image that comes with all the software libraries and tools necessary to take advantage of InfiniBand through supported ML and HPC frameworks. While you are not required to use this image to use InfiniBand networking, we strongly recommend the `ubuntu22.04-nvidia-sxm-docker:latest` image for the easiest possible setup. Learn more about [images](../../compute/images/overview).
:::info
To support full-performance distributed training on Crusoe’s hypervisor/virtualisation stack, some NCCL configuration changes are required.
:::
### `NCCL Configuration`
Crusoe’s updated hypervisor stack changes the PCI address of devices within the virtual machines. This requires an update to the NCCL XML topology file. The XML file is included by default in Crusoe’s curated image, you can set the following environment variable in `/etc/nccl.conf`.
```sh
NCCL_TOPO_FILE=
```
The `*-nvidia-sxm-docker` images also comes with a service unit file called `crusoe_nccl_topo.service` which sets the `NCCL_TOPO_FILE` environmental variable. If a custom topology file location is used (for example, to customize the topology file), the service should be disabled by running `systemctl disable crusoe_nccl_topo.service`. The current state of the service may be queried by running `systemctl status crusoe_nccl_topo.service`.
For NCCL to correctly detect the PCIe topology, the following environment variables must be set in `/etc/nccl.conf`.
```sh
NCCL_IB_MERGE_VFS=0
```
The NCCL_IB_HCA configuration must be modified to exclude the Ethernet device `mlx5_0`.
```sh
NCCL_IB_HCA=^mlx5_0:1
```
When using versions of `HPC-X` older than `2.18`, the following argument must be used.
```sh
NCCL_IBEXT_DISABLE=1
```
If you are not using this image, or want to run containers, you must also set all these environment variables:
```sh
FROM ...
ENV NCCL_TOPO_FILE=/path/to/nccl_topo.xml
ENV NCCL_IB_MERGE_VFS=0
ENV NCCL_IB_HCA=^mlx5_0:1
```
### `NCCL_TOPO_FILE`
Provided below are the NCCL Topology files:
| Instance Type | NCCL Topology File |
| ---------------------- | ---------------------------------------------------------- |
| `a100-80gb-sxm-ib.8x` | [a100-80gb-sxm-ib.8x](./assets/topo/a100-80gb-sxm-ib.xml) |
| `h100-80gb-sxm-ib.8x` | [h100-80gb-sxm-ib.8x](./assets/topo/h100-80gb-sxm-ib.xml) |
| `h200-141gb-sxm-ib.8x` | [h200-80gb-sxm-ib.8x](./assets/topo/h200-141gb-sxm-ib.xml) |
## InfiniBand Networks
InfiniBand Networks are a logical representation of the physical InfiniBand fabric.
### InfiniBand Network limitations
You are limited to a maximum of five InfiniBand partitions on the same InfiniBand network.
## Listing InfiniBand Networks and Partitions
**CLI:**
Use the `networking ib-networks list` and `networking ib-partitions list` commands to list networks and partitions.
```sh
crusoe networking ib-networks list
```
```sh
crusoe networking ib-partitions list
```
The IDs will be used when attaching a VM to a partition.
**UI:**
To view InfiniBand networks and partitions via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Infiniband](https://console.crusoecloud.com/networking/infiniband)** in the left nav.
2. View the InfiniBand networks and partitions.
**Terraform:**
To list existing IB networks using Terraform, the following code snippet can be used to populate a Terraform data source using the Crusoe Terraform provider.
```hcl
# list ib networks
data "crusoe_ib_networks" "ib_networks" {}
output "crusoe_ib" {
value = data.crusoe_ib_networks.ib_networks
}
```
Listing IB partitions is not currently supported in the Crusoe Terraform provider.
## Creating InfiniBand Partitions
**CLI:**
Use the `networking ib-partitions create` command to create a new partition.
```sh
crusoe networking ib-partitions create \
--name my-new-partition \
--ib-network-id uuid-of-network
```
**UI:**
To create an IB partition via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Infiniband](https://console.crusoecloud.com/networking/infiniband)** in the left nav.
2. Click **Create Partition**.
3. Input a name for the IB partition.
4. Select the IB network to create the IB partition in.
5. Click **Create**.
**Terraform:**
The following is intended to help get you started using Terraform to provision and InfiniBand Partition and attach an InfiniBand partition to a VM in Crusoe Cloud.
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates an InfiniBand partition:
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
# list IB networks
data "crusoe_ib_networks" "ib_networks" {}
output "crusoe_ib" {
value = data.crusoe_ib_networks.ib_networks
}
# create an IB partition to deploy VMs in
resource "crusoe_ib_partition" "my_partition" {
name = "my-ib-partition"
# available IB network IDs can be listed by using the output
# above. alternatively, they can be obtain with the CLI by
# crusoe networking ib-networks list
# copy and paste the network ID that has capacity
ib_network_id = data.crusoe_ib_networks.ib_networks.ib_networks[0].id
}
```
`name` and `ib_network_id` are required when creating the resource `crusoe_ib_partition`
`name` can only include lowercase ascii characters, numbers and `-`.
`ib_network_id` is a string which represents the UUID of an existing InfiniBand Network from the output of `crusoe networking ib-networks list`
## Update an existing InfiniBand Partition
**CLI:**
Updating Infiniband partitions is currently unsupported in the Crusoe Cloud CLI.
**UI:**
Updating Infiniband partitions is currently unsupported in the Crusoe Cloud Console.
**Terraform:**
To update an IB network using the Crusoe Terraform provider, you can change the fields of an existing IB network and run `terraform apply`. The Crusoe Terraform provider will apply the changes to the IB network.
```hcl
# Update the existing IB partition with the new network ID
resource "crusoe_ib_partition" "my_partition" {
name = "my-ib-partition" -> "new-ib-partition-name"
ib_network_id = "ib-network-id" -> "new-ib-network-id"
project_id = "project-id" -> "new-project-id"
}
```
Currently, only the "name", "ib_network_id", or "project_id" can be updated. These fields cannot be updated in-place. As a result, modifying any of these fields will replace the resource rather than an in-place update.
## Deleting an InfiniBand Partition
:::info
**Warning:** deleting an InfiniBand partition is a permanant action that will require re-creation of the partition to recover.
:::
**CLI:**
Infiniband partitions can be deleted in the CLI using the `networking ib-partitions delete ` command.
**UI:**
To delete an Infiniband partition via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Infiniband](https://console.crusoecloud.com/networking/infiniband)** in the left nav.
2. Navigate to the row of the Infiniband partition you want to delete.
3. Click the trash can icon on the far right side of the row.
4. Click **Confirm**.
**Terraform:**
An Infiniband partition can be deleted by using the `terraform destroy` command provided by the Terraform CLI tool.
## Launching Instances in an InfiniBand Parition
**CLI:**
Use the `compute vms create` command to create a new VM, passing in the `--ib-partition-id`:
```sh
crusoe compute vms create \
--name infiniband-test \
--location us-east1-a \
--type a100-80gb-sxm-ib.8x \
--image ubuntu22.04-nvidia-sxm-docker:latest \
--ib-partition-id uuid-of-partition \
...
```
**UI:**
To connect a VM to an IB partition, you must create the VM and attach it to the appropriate InfiniBand Network and Partition:
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Click **Create instance**.
3. Select the desired instance type and location, ensuring that the instances and location support InfiniBand.
4. Select the desired InfiniBand Network.
5. Select the desired InfiniBand Partition, or create a new InfiniBand Partition.
6. Continue through the rest of the flow.
7. Click **Create**.
**Terraform:**
Copy and paste the code below in a text-editor of your choice and name the file `main.tf`. The example below creates an Infiniband partition attached to a VM along with a disk attached:
```hcl
// Crusoe Provider
terraform {
required_providers {
crusoe = {
source = "registry.terraform.io/crusoecloud/crusoe"
}
}
}
locals {
my_ssh_key = file("~/.ssh/id_rsa.pub")
}
# attached storage disk
resource "crusoe_storage_disk" "data_disk" {
name = "data-disk"
size = "1TiB"
location = "us-east1-a"
}
# list IB networks
data "crusoe_ib_networks" "ib_networks" {}
output "crusoe_ib" {
value = data.crusoe_ib_networks.ib_networks
}
# create an IB partition to deploy VMs in
resource "crusoe_ib_partition" "my_partition" {
name = "my-ib-partition"
ib_network_id = data.crusoe_ib_networks.ib_networks.ib_networks[0].id
}
# create multiple VMs, all in the same Infiniband partition
resource "crusoe_compute_instance" "my_vm" {
count = 3
name = "ib-vm-${count.index}"
type = "h100-80gb-sxm-ib.8x" // other option: a100-80gb-sxm-ib.8x
location = "us-east1-a" # IB currently only supported at us-east1-a
image = "ubuntu22.04-nvidia-sxm-docker:latest" # recommended IB image
ssh_key = local.my_ssh_key
host_channel_adapters = [
{
ib_partition_id = crusoe_ib_partition.my_partition.id
}
]
disks = [
// disk attached at startup
{
id = crusoe_storage_disk.data_disk.id
attachment_type = "data"
mode = "read-write" // other option: "read-only"
}
]
}
```
## Updating the IB partition on a VM
**CLI:**
Use the `compute vms update` on a stopped VM to update the `--ib-partition-id`.
```sh
crusoe compute vms update \
--ib-partition-id uuid-of-new-partition
```
**UI:**
To switch IB networks or partitions in the [console](https://console.crusoecloud.com):
1. From the console, select **Compute** > **[Instances](https://console.crusoecloud.com/compute/instances)** in the left nav.
2. Select the appropriate instance.
3. Stop the instance if it isn't already stopped.
4. Pick the appropriate IB network and partition from the dropdowns.
5. Start the instance and verify that it's connected to the chosen IB network and partition.
---
# Validating Infiniband Performance
## Validating IB networking Performance - NVIDIA Collective Communication Library
You can utilize NVIDIA's Collective Communications Library (NCCL) to validate the performance of the IB networking stack once at least 2 IB supported instance types are launched within an IB partition, following these steps:
1. Setup passwordless SSH between the two instances by first SSH connecting into one of the instances and issuing a `ssh-keygen -t ed25519`. Accept the defaults and then copy the contents of the `./ssh/ed25519.pub` to both instances' `.ssh/authorized_keys` file.
2. On each instance git clone the [NVIDIA/nccl-tests](https://github.com/NVIDIA/nccl-tests) repo and compile the binaries according to the repo's instructions.
3. One of the instances create a `hostfile` referencing the private IP of each instance and `slots=8` which represents that each instance supports 8 ranks (8 GPUs per instance). As an example hostfile below:
```sh
172.27.21.150 slots=8
172.27.22.114 slots=8
```
4. Run the below NCCL all reduce command with directives corresponding to your instance. We have chosen UCX for the transport protocol which is part of the nccl-rdma-sharp plugins. We recommend using UCX for optimized performance.
```sh
#!/bin/bash
. /opt/hpcx/hpcx-init.sh
hpcx_load
mpirun -np 16 -N 8 -x NCCL_DEBUG=INFO -hostfile hostfile \
--bind-to none -mca btl tcp,self -mca coll_hcoll_enable 0 \
-x NCCL_IB_AR_THRESHOLD=0 -x NCCL_IB_PCI_RELAXED_ORDERING=1 \
-x NCCL_IB_SPLIT_DATA_ON_QPS=0 -x NCCL_IB_QPS_PER_CONNECTION=2 -x CUDA_DEVICE_ORDER=PCI_BUS_ID \
-x PATH -x LD_LIBRARY_PATH -x NCCL_IB_HCA=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1 \
/home/ubuntu/nccl-tests/build/all_reduce_perf -b 8 -e 2G -f 2 -t 1 -g 1 -c 1 -n 100
```
With `NCCL_DEBUG=INFO` set you will get a verbose output of the NCCL communication layer bootstraping the connection as well as reporting the connection topology and specific environment variables used. You can confirm a few additional key points to ensure optimum performance:
1. Ensure that each rank reports back that the correct and NCCL topology file is being used. This is a global environment variable that is set by Crusoe's curated images to ensure that the correct CPU affinity, GPUs, and IB HCAs are in the expected topology.
```
aragab-ib7:24224:24290 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/crusoe/nccl_topo/h100-80gb-sxm-ib.xml
aragab-ib7:24223:24291 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/crusoe/nccl_topo/h100-80gb-sxm-ib.xml
aragab-ib7:24227:24292 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/crusoe/nccl_topo/h100-80gb-sxm-ib.xml
```
2. Crusoe's curated images load the `nvidia_peermem` module by default which will enable GPUDirectRDMA. In the NCCL debug out you can confirm that by ensuring you have:
```
...
aragab-ib7:24224:24290 [5] NCCL INFO Channel 04/0 : 13[5] -> 4[4] [send] via NET/UCX/0/GDRDMA
aragab-ib7:24224:24290 [5] NCCL INFO Channel 12/0 : 13[5] -> 4[4] [send] via NET/UCX/0/GDRDMA
...
```
or
```
...
aragab-ib10:12931:12936 [6] NCCL INFO Channel 06/0 : 166[6] -> 150[6] [send] via NET/IBext/2/GDRDMA
aragab-ib10:12931:12936 [6] NCCL INFO Channel 22/0 : 166[6] -> 150[6] [send] via NET/IBext/2/GDRDMA
...
```
Depending on whether you are using the UCX or IBext plugins provided by the NVIDIA HPCX libraries.
The results of the 2 instance NCCL tests should look similar to the below output for each instance type:
**a100-80gb-sxm-ib.8x:**
```
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
8 2 float sum -1 35.11 0.00 0.00 0 34.63 0.00 0.00 0
16 4 float sum -1 34.29 0.00 0.00 0 34.70 0.00 0.00 0
32 8 float sum -1 34.90 0.00 0.00 0 34.66 0.00 0.00 0
64 16 float sum -1 33.03 0.00 0.00 0 33.21 0.00 0.00 0
128 32 float sum -1 33.58 0.00 0.01 0 33.44 0.00 0.01 0
256 64 float sum -1 33.81 0.01 0.01 0 33.70 0.01 0.01 0
512 128 float sum -1 36.15 0.01 0.03 0 36.11 0.01 0.03 0
1024 256 float sum -1 37.18 0.03 0.05 0 37.21 0.03 0.05 0
2048 512 float sum -1 38.58 0.05 0.10 0 38.08 0.05 0.10 0
4096 1024 float sum -1 39.94 0.10 0.19 0 39.50 0.10 0.19 0
8192 2048 float sum -1 42.07 0.19 0.37 0 41.52 0.20 0.37 0
16384 4096 float sum -1 44.84 0.37 0.69 0 42.15 0.39 0.73 0
32768 8192 float sum -1 49.97 0.66 1.23 0 42.10 0.78 1.46 0
65536 16384 float sum -1 50.16 1.31 2.45 0 52.09 1.26 2.36 0
131072 32768 float sum -1 64.25 2.04 3.83 0 62.40 2.10 3.94 0
262144 65536 float sum -1 68.07 3.85 7.22 0 66.84 3.92 7.35 0
524288 131072 float sum -1 74.99 6.99 13.11 0 72.28 7.25 13.60 0
1048576 262144 float sum -1 84.25 12.45 23.34 0 83.38 12.58 23.58 0
2097152 524288 float sum -1 109.4 19.18 35.95 0 125.8 16.67 31.25 0
4194304 1048576 float sum -1 141.9 29.56 55.43 0 156.0 26.89 50.42 0
8388608 2097152 float sum -1 203.3 41.27 77.37 0 201.2 41.68 78.16 0
16777216 4194304 float sum -1 419.8 39.97 74.94 0 270.1 62.11 116.45 0
33554432 8388608 float sum -1 493.4 68.01 127.51 0 493.6 67.98 127.46 0
67108864 16777216 float sum -1 910.9 73.67 138.14 0 911.2 73.65 138.09 0
134217728 33554432 float sum -1 1374.7 97.63 183.06 0 1371.5 97.86 183.48 0
268435456 67108864 float sum -1 2725.5 98.49 184.67 0 2672.7 100.44 188.32 0
536870912 134217728 float sum -1 5310.2 101.10 189.57 0 5297.2 101.35 190.03 0
1073741824 268435456 float sum -1 10475 102.50 192.19 0 10455 102.70 192.57 0
2147483648 536870912 float sum -1 20656 103.96 194.93 0 20685 103.82 194.66 0
```
**h100-80gb-sxm-ib.8x:**
```
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
8 2 float sum -1 31.73 0.00 0.00 0 31.70 0.00 0.00 0
16 4 float sum -1 31.58 0.00 0.00 0 31.57 0.00 0.00 0
32 8 float sum -1 31.74 0.00 0.00 0 31.66 0.00 0.00 0
64 16 float sum -1 30.27 0.00 0.00 0 30.20 0.00 0.00 0
128 32 float sum -1 30.53 0.00 0.01 0 30.45 0.00 0.01 0
256 64 float sum -1 40.39 0.01 0.01 0 30.70 0.01 0.02 0
512 128 float sum -1 32.01 0.02 0.03 0 31.41 0.02 0.03 0
1024 256 float sum -1 32.66 0.03 0.06 0 32.40 0.03 0.06 0
2048 512 float sum -1 33.82 0.06 0.11 0 33.48 0.06 0.11 0
4096 1024 float sum -1 35.14 0.12 0.22 0 34.90 0.12 0.22 0
8192 2048 float sum -1 36.06 0.23 0.43 0 35.70 0.23 0.43 0
16384 4096 float sum -1 38.19 0.43 0.80 0 37.51 0.44 0.82 0
32768 8192 float sum -1 39.62 0.83 1.55 0 38.80 0.84 1.58 0
65536 16384 float sum -1 41.42 1.58 2.97 0 40.81 1.61 3.01 0
131072 32768 float sum -1 45.61 2.87 5.39 0 45.04 2.91 5.46 0
262144 65536 float sum -1 53.40 4.91 9.20 0 53.37 4.91 9.21 0
524288 131072 float sum -1 60.98 8.60 16.12 0 60.53 8.66 16.24 0
1048576 262144 float sum -1 73.81 14.21 26.64 0 72.83 14.40 27.00 0
2097152 524288 float sum -1 103.1 20.34 38.13 0 102.7 20.42 38.28 0
4194304 1048576 float sum -1 123.0 34.11 63.96 0 119.5 35.09 65.80 0
8388608 2097152 float sum -1 137.3 61.10 114.56 0 123.4 67.98 127.45 0
16777216 4194304 float sum -1 195.6 85.78 160.83 0 192.1 87.35 163.78 0
33554432 8388608 float sum -1 544.4 61.64 115.58 0 419.7 79.95 149.90 0
67108864 16777216 float sum -1 468.6 143.22 268.53 0 469.1 143.06 268.23 0
134217728 33554432 float sum -1 779.9 172.09 322.67 0 776.6 172.83 324.05 0
268435456 67108864 float sum -1 1448.4 185.33 347.49 0 1447.9 185.40 347.63 0
536870912 134217728 float sum -1 2881.9 186.29 349.29 0 2879.8 186.43 349.56 0
1073741824 268435456 float sum -1 5674.6 189.22 354.78 0 5676.4 189.16 354.67 0
2147483648 536870912 float sum -1 11152 192.56 361.06 0 11127 193.00 361.87 0
```
If you are unable to reach your desired level of performance, please [contact support](../../resources/support.md).
---
# InfiniBand Metrics
Crusoe Cloud provides out-of-the-box InfiniBand (IB) metrics to help you monitor network performance, identify failures, and optimize utilization. These metrics are collected and published in 5-minute intervals, and are retained for 30 days. You can view IB metrics directly within the Crusoe Console under the VM Metrics view, or access them via our Prometheus-compatible query API. For detailed instructions on connecting via the API or Grafana, see the [VM Telemetry page](https://docs.crusoecloud.com/compute/virtual-machines/vm-telemetry).
| **Metrics** | **Definition** | **Suggested Query** |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| Line Rate | The maximum theoretical data transfer speed of the InfiniBand link, measured in Gigabits per second (Gb/s). | `crusoe_ib_port_line_rate` |
| InfiniBand Throughput Tx (bytes per second) | The rate of data transmitted from the port, measured in bytes per second. | `rate(crusoe_ib_port_throughput_tx[5m])` |
| InfiniBand Throughput Rx (bytes per second) | The rate of data received by the port, measured in bytes per second. | `rate(crusoe_ib_port_throughput_rx[5m])` |
| InfiniBand Throughput Tx (Packets per second) | The rate of packets transmitted from the port, measured in packets per second. | `rate(crusoe_ib_port_packets_tx[5m])` |
| InfiniBand Throughput Rx (Packets per second) | The rate of packets received by the port, measured in packets per second. | `rate(crusoe_ib_port_packets_rx[5m])` |
| InfiniBand Transmit Wait | The rate at which packets had to wait before being transmitted from the port, indicating a congestion or scheduling issue. | `rate(crusoe_ib_port_tx_wait[5m])` |
| InfiniBand Link Downed | The rate of the InfiniBand link transitioned from an active state to a link-down state. | `rate(crusoe_ib_port_link_downed[5m])` |
| InfiniBand Link Error Recovery | The rate of the link underwent an error recovery process to attempt to restore a healthy link state. | `rate(crusoe_ib_port_link_error_recovery[5m])` |
| InfiniBand Port Constraint Errors Tx | The rate of errors transmitted due to link protocol or connectivity constraints, measured in packets per second. | `rate(crusoe_ib_port_constraint_error_tx[5m])` |
| InfiniBand Port Constraint Errors Rx | The rate of errors received due to link protocol or connectivity constraints, measured in packets per second. | `rate(crusoe_ib_port_constraint_error_rx[5m])` |
| InfiniBand Port Errors Rx | The rate of received packets with errors at the port level, typically indicating issues like bad Cyclic Redundancy Check (CRC) errors. | `rate(crusoe_ib_port_error_rx[5m])` |
| InfiniBand Symbol Errors | The rate of low-level physical error bits received where the receiver detected an invalid data symbol, measured in bits per second. | `rate(crusoe_ib_port_bits_error[5m])` |
| InfiniBand Remote Physical Errors (API only) | The rate of remote physical errors received at the port. | `rate(crusoe_ib_port_rcv_remote_physical_errors[5m])` |
| InfiniBand Transmit Discards (API only) | The rate of packets discarded during transmission. | `rate(crusoe_ib_port_xmit_discard[5m])` |
| InfiniBand Local Link Integrity Errors (API only) | The rate of local link integrity errors detected on the port. | `rate(crusoe_ib_local_link_integrity_errors[5m])` |
---
# Overview
Crusoe Cloud currently offers L4 passthrough load balancers through our managed load balancers, allowing you to distribute TCP traffic across one or more backends. These load balancers can be used to distribute inference traffic across your GPU fleet, provide ingress for dashboards and services used to monitor model training runs or front self-managed cluster control planes.
:::info
This capability is currently in Limited Availability, and may not be visible to your organization by default. To request access, please [contact support](../../resources/support.md).
:::
## Details
Load balancers on Crusoe Cloud are regional resources and are tied to a specific VPC network. Each load balancer receives a static public IPv4 address and be configured to listed for traffic on one or more ports. Any traffic received to these ports can be routed to one or more backends.
### Key Concepts
#### Backends and Health Checks
Each load balancer backend is represented by a tuple of private IPv4 address:port. These backends typically represent services running on Crusoe Cloud virtual machines, either directly or via a pod orchestrated by a Kuberentes cluster.
The health of each of these backends is monitored by a configurable TCP health check. Any backends that are detected as non-responsive are marked as 'Offline', with traffic no longer routed to them by the load balancer. The table below provides an overview of the fields that can be configured for the health check.
| Field | Description | Default Value |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------- |
| `Timeout` | The maximum amount of time, in seconds, that the load balancer will wait for a healthy response from a backend. | `5 seconds` |
| `Interval` | The frequency, in seconds, at which the load balancer will send health check probes to a backend server. | `5 seconds` |
| `Success Count` | The number of consecutive successful health checks required for a backend to be brought back online | `3` |
| `Failure Count` | The number of consecutive failed health checks that will cause a backend server to be marked as unhealthy and taken out of rotation. | `2` |
#### Protocols
Only TCP traffic load balancing is supported currently.
#### Load Balancing Algorithms
Traffic is distributed across each load balancer using IP-hash based load balancing based on the source IP of inbound traffic. This is not currently configurable.
#### Firewall Rules
Firewall rules are not currently created by default when backends are added to a load balancer. To ensure that your backends receive inbound traffic on the desired ports, please create the appropriate [firewall rules](../firewall-rules/managing-firewall-rules.mdx).
Firewall rules must also be applied directly to the backends that are attached to the load balancer. We do not currently support specifying firewall rules directly for the load balancer resource. For example, if you create a Kubernetes service object in a Kubernetes cluster that is deployed in `default-subnet-eu-iceland1-a`, you must create an ingress firewall rule. This rule should allow inbound traffic from any source IP ( `0.0.0.0/0` ) to the destination ( `default-subnet-eu-iceland1-a` ) subnet and the `nodePort` of the service object.
#### Kubernetes LoadBalancer Service Integration
We support a dedicated [Load Balancer controller](https://github.com/crusoecloud/crusoe-load-balancer-controller-helm-charts) that creates and manages Load Balancers in your project / organization when LoadBalancer services are created within your Kubernetes cluster. Installation instructions are outlined [here](https://github.com/crusoecloud/crusoe-load-balancer-controller-helm-charts).
---
# Managing Load Balancers
# Managing your networks and subnets
## Creating a new Load Balancer
**CLI:**
Use the `networking load-balancers create` command to create a new load balancer.
```sh
crusoe networking load-balancers create \
--name my-new-lb \
--location us-east1-a \
--vpc-network-id \
--protocol tcp \
--listener 80,172.1.1.1:80,172.1.1.2:80
```
**UI:**
To create a load balancer via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Load Balancers](https://console.crusoecloud.com/networking/load-balancers)** in the left nav.
2. Click **Create Load Balancer**.
3. Input a name, location, VPC network, listen port(s), and other required fields for the load balancer.
4. Click **Create Load Balancer**.
## Viewing all Load Balancers
**CLI:**
Use the `networking load-balancers list` command to list existing load balancers.
```sh
crusoe networking load-balancers list
```
**UI:**
To view load balancers via the [console](https://console.crusoecloud.com), select **Networking** > **[Load Balancers](https://console.crusoecloud.com/networking/load-balancers)** in the left nav.
## Update an existing Load Balancer
**CLI:**
Load balancers may be updated by using the `networking load-balancers update ` command. Note that the list of backends associated with each listen port is declarative. You will have to specify the complete list of backends when updating your load balancer.
```sh
crusoe networking load-balancers update \
--listener 80,172.1.1.1:80,172.1.1.2:80,127.0.0.1:80 \
--listener 443,172.1.1.1:443,172.1.1.2:443,127.0.0.1:443
--health-check 10,5,5,1
```
**UI:**
To update an existing load balancer via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Load Balancers](https://console.crusoecloud.com/networking/load-balancers)** in the left nav.
2. Select the load balancer you want to edit.
3. Use the **Edit Health Checks** field to modify health parameters, or modify destinations via the **Destinations** table.
## Deleting a Load Balancer
**CLI:**
Load Balancers can be deleted in the CLI using the `networking load-balancers delete ` command.
**UI:**
To delete a load balancer via the [console](https://console.crusoecloud.com):
1. From the console, select **Networking** > **[Load Balancers](https://console.crusoecloud.com/networking/load-balancers)** in the left nav.
2. Click on the load balancer you want to delete.
3. Select **Delete** and type in the resource name as confirmation.
If you are having issues working with your load balancers, please [contact support](../../resources/support.md).
---
# Load Balancer Metrics
Load Balancer Metrics provide comprehensive insights into the throughput of your load balancers. These metrics help you monitor traffic patterns, analyze connection behavior, and understand data flow through your load balancers. The following key metrics are collected in 30-second intervals for a period of 30 days for each load balancer.
| **Metrics** | **Definition** | **Suggested Query** |
| :--------------- | :-------------------------------------------------------------------- | :------------------------------- |
| Outbound Packets | The number of packets transmitted by the load balancer. | rate(crusoe_elb_out_packets[5m]) |
| Inbound Packets | The number of packets received by the load balancer. | rate(crusoe_elb_in_packets[5m]) |
| Bytes Out | The total bytes transmitted by the load balancer. | rate(crusoe_elb_bytes_out[5m]) |
| Bytes In | The total bytes received by the load balancer. | rate(crusoe_elb_bytes_in[5m]) |
| Active Flows | The number of currently active connections through the load balancer. | crusoe_elb_active_flows |
| New Flows | The number of new connections established through the load balancer. | rate(crusoe_elb_new_flows[5m]) |
Load Balancer Metrics are available by default for all load balancers. They're accessible via the Console, a PromQL API, Telemetry Conduit, and Crusoe MCP.
**Console:** Navigate to **Networking** > **[Load Balancers](https://console.crusoecloud.com/networking/load-balancers)** in the left navigation, select the load balancer you want to inspect, then select the **Metrics** tab.
**PromQL API:** By default, queries return metrics for all load balancers within a project. To query metrics for a specific load balancer, add a label selector to your query:
```
crusoe_elb_active_flows{elb_name="my-load-balancer"}
```
For token generation and instructions on querying via API, Grafana, or Telemetry Conduit, see [Get started](../../command-center/get-started.mdx).
---
# Setting up IMEX
## Setting up and Validating IMEX
[NVIDIA's IMEX service](https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/overview.html) supports GPU memory export and import (NVLink P2P) and shared memory operations across OS domains in a NVLink multi-node deployment. On Crusoe Cloud this is relevant for rack scale solutions like the GB200.
IMEX is facilitated by the daemon called nvidia-imex.service. What this does is:
- It manages the GPU sharing lifecycle & runs on the compute nodes.
- Operates below the application level and outside the CUDA/NCCL stack.
- Communicates via TCP/IP or gRPC connections.
This page provides a step by step guide to setting up and validating that IMEX is working correctly.
### Step 1: Check IMEX service status to ensure it is enabled
`nvidia-imex-ctl` & `nvidia-imex` binaries are already available on the GB200 VM Image. To check the status, run the following command.
```sh
sudo systemctl status nvidia-imex
```
If the service is not enabled, you can enable it using the following command.
```sh
sudo systemctl enable nvidia-imex
```
### Step 2: Create node_config with compute tray IP address
Create the file `/etc/nvidia-imex/nodes_config.cfg` and add all the private IPs of the GB200 VMs in the NVLink domain. An example format for the file is shown below
```
172.27.58.70
172.27.58.71
172.27.58.72
172.27.58.73
```
After updating the config file with all VM IPs, restart the IMEX service
```sh
sudo systemctl stop nvidia-imex
sudo systemctl start nvidia-imex
```
_Note:_ The IMEX config file is read as part of the IMEX service startup process. If you change the config file options, for the new settings to take effect you will need to restart the IMEX service.
### Step 3: Create IMEX Channels
[IMEX channels](https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/imexchannels.html) are a GPU driver feature that allows for user-based memory isolation in a multi-user environment within an IMEX domain.
Create a channel that allows user based memory isolation in the multi-node setup.
```sh
# Get the major number & create the default channel. In the following example output, 234 is the major number
$ grep nvidia-caps-imex-channels /proc/devices
234 nvidia-caps-imex-channels
# Create the channel. THIS WILL HAVE TO BE CREATED EVERYTIME AT REBOOT due /dev reset.
sudo mkdir /dev/nvidia-caps-imex-channels/
sudo mknod /dev/nvidia-caps-imex-channels/channel0 c 0
# FOR CREATING A PERSISTENT CHANNEL POST REBOOT
$ vi /etc/modprobe.d/nvidia.conf$
options nvidia NVreg_CreateImexChannel0=1 # <== ADD THIS OPTION AND SAVE
# REGENERATE initramfs
sudo update-initramfs -u
sudo reboot
```
### Step 4: Check whole rack connectivity
Check the connectivity of the whole rack using
```sh
nvidia-imex-ctl -N
```
If **'C' (Connected)** appears for all nodes, then the rack connectivity check has passed.
If **'I' (Invalid)** if there is an error. Run `sudo journalctl -u nvidia-imex` to view the IMEX logs.
If all nodes are connected the following table will be returned
```sh
Nodes From\To 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
0 C C C C C C C C C C C C C C C C C C
1 C C C C C C C C C C C C C C C C C C
2 C C C C C C C C C C C C C C C C C C
3 C C C C C C C C C C C C C C C C C C
4 C C C C C C C C C C C C C C C C C C
5 C C C C C C C C C C C C C C C C C C
6 C C C C C C C C C C C C C C C C C C
7 C C C C C C C C C C C C C C C C C C
8 C C C C C C C C C C C C C C C C C C
9 C C C C C C C C C C C C C C C C C C
10 C C C C C C C C C C C C C C C C C C
11 C C C C C C C C C C C C C C C C C C
12 C C C C C C C C C C C C C C C C C C
13 C C C C C C C C C C C C C C C C C C
14 C C C C C C C C C C C C C C C C C C
15 C C C C C C C C C C C C C C C C C C
16 C C C C C C C C C C C C C C C C C C
17 C C C C C C C C C C C C C C C C C C
```
For any issues `sudo journalctl -u nvidia-imex` can be run to view the IMEX logs.
---
# Overview
# Crusoe Managed Kubernetes (CMK): Overview
Crusoe Managed Kubernetes (CMK) allows you to spin up Kubernetes clusters with fully-managed control planes through our UI, CLI, API and Terraform provider. We manage the availability, scaling and lifecycle of cluster control plane nodes, offer simple primitives to automate node registration and provide out of the box installation of required GPU and network drivers, enabling you to build and scale your AI workloads with minimal overhead.
## Key Capabilities
- **Managed Clusters:** You may provision one or more clusters in your project. Each cluster consists of a minimum of 3 control plane nodes. Control plane nodes are distributed across different physical hosts (at a minimum) to provide a high availability posture.
- **Node Pools:** Node pools allow you to group one or more Crusoe Cloud instances and associate them to a specific cluster control plane, to function as worker nodes. You may provision one or more node pools in the context of a specific cluster.
- **Add-ons:** When provisioning a cluster, you may choose to install one or more add-ons that install additional functionality in the cluster. These include the [Nvidia GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html), [Nvidia Network Operator](https://docs.nvidia.com/networking/display/cokan10/network+operator) and the [Crusoe Container Storage Interface](https://github.com/crusoecloud/crusoe-csi-driver).
- **Autoclusters:** AutoClusters enhances the resilience of your CMK workloads by automatically detecting and resolving common hardware failures. By enabling AutoClusters, you can minimize downtime and reduce the need for manual intervention, ensuring higher effective utilization for your clusters.
## Pricing
CMK clusters are billed at a rate of $0.10 per cluster-hour. For more information, please visit our [website](https://crusoe.ai/cloud/).
---
# Shared Responsibility Model
Crusoe Managed Kubernetes (CMK) is a managed Kubernetes service purpose-built for GPU-accelerated and AI/ML workloads. Operating a production Kubernetes environment is a shared effort — Crusoe is responsible for the underlying infrastructure and control plane, while you are responsible for your workloads and applications. The layers in between involve shared ownership, and this document describes the specific boundaries at each layer.
## Responsibility by Layer
At a high level, Crusoe manages the infrastructure and control plane. You manage your workloads and applications. The layers in between are shared, with clear ownership at each component. Where a category appears in multiple columns below, individual components within that category have different owners — the detailed sections that follow break down which parts are Crusoe-managed and which are shared or customer-owned.
| | Crusoe | Shared | Customer |
| ---------------------- | :----: | :----: | :------: |
| **Workloads** | | | ● |
| **Application Config** | | | ● |
| **Access & RBAC** | | ● | |
| **Add-Ons** | | ● | |
| **Worker Nodes** | | ● | |
| **Networking** | | ● | |
| **Storage** | | ● | |
| **Control Plane** | ● | | |
| **Infrastructure** | ● | | |
## What This Means at Each Layer
### Infrastructure
We own the physical environment. You never touch hardware.
**Crusoe manages:** Physical hardware (GPUs, CPUs), data centers, power, cooling, hypervisor / bare-metal provisioning, and InfiniBand RDMA networking fabric.
### Control Plane
We run and upgrade the Kubernetes control plane. You choose the version and tell us when to upgrade.
**Crusoe manages:** kube-apiserver, etcd, kube-scheduler, kube-controller-manager. We handle provisioning (minimum 3 nodes, distributed for HA), availability, scaling, patching, and Kubernetes version upgrades.
**You manage:** Kubernetes version selection at cluster creation. When you are ready to upgrade, you request the upgrade from Crusoe and we handle it.
### Worker Nodes
Crusoe provides the base machine images for worker nodes. You decide how many nodes to run, how they are organized into node pools, and when to apply updates.
**Crusoe manages:** Worker node base OS images and base OS configuration.
**Shared:** OS updates and patching (Crusoe provides updated images; you apply them by cycling nodes in your node pools). Cluster Autoscaler (Crusoe provides a CMK-compatible build; you configure min/max and deploy). AutoClusters (Crusoe detects hardware failures and remediates when allowed; you configure settings).
**You manage:** Node pool creation, configuration, sizing, manual scaling, and deletion (required before cluster deletion).
### Networking
Crusoe provides the cluster networking layer, load balancers, and the firewall rules required for cluster operation. You configure your network topology, application traffic rules, and any additional firewall rules for your workloads.
**Crusoe manages:** CNI (Cilium) — installation and default configuration. Load balancers (L4 passthrough). Cluster and node pool firewall rules required for cluster operation (created on cluster creation, updated as necessary, removed on cluster deletion).
**Shared:** Firewall configuration (Crusoe creates and manages rules required for cluster functionality; you create and manage rules for exposing your applications and workloads).
**You manage:** VPC / subnet configuration, Pod CIDR, Subnet Mask and Service CIDR (set at cluster creation), Kubernetes Services and Ingress objects, network policies, and application-level firewall rules.
### Storage
We provide the underlying storage infrastructure. You define how storage is allocated and consumed by your workloads.
**Crusoe manages:** Persistent Disk block storage infrastructure, Shared Disk / NFS (VAST Data) backend. NFS enablement must be requested from Crusoe Support per project.
**Shared:** Crusoe CSI Driver (Crusoe provides the Helm chart; you install and configure).
**You manage:** StorageClass definitions, PersistentVolumeClaim definitions, and any storage migration steps (e.g., VirtioFS to NFS).
### Add-Ons and Operators
Crusoe provides a set of core cluster add-ons that extend cluster functionality for GPU-accelerated workloads. You may opt in to these add-ons at cluster creation or install them later via Helm. Any additional operators or charts you bring are yours to manage.
**Crusoe manages:** Installation of core cluster add-ons, including Cilium, the NVIDIA GPU Operator, and the NVIDIA Network Operator (required for InfiniBand-enabled instances). Crusoe manages critical upgrades to these add-ons.
**Shared:** Add-on configuration (Crusoe provides a reference configuration for GPU and Network Operators; you may customize settings such as driver versions to suit your workloads).
**You manage:** All other third-party Helm charts and operators. If you install your own GPU or Network Operator outside of the Crusoe-provided add-ons, or significantly modify the Crusoe-provided configuration, that add-on becomes Customer-Owned.
### Identity, Access, and Security
We provide the IAM framework and container registry. You manage who has access and what they can do.
**Crusoe manages:** Project-level IAM (Admin / Editor / Reader roles). Crusoe Container Registry (CCR) infrastructure.
**Shared:** kubeconfig generation (Crusoe generates; you download and manage locally). CCR token rotation (Crusoe provides CronJob Helm chart; you install and configure). OIDC configuration (you provide the identity provider configuration; Crusoe applies it to cluster components such as kube-apiserver).
**You manage:** Organization member roles, Crusoe API keys, Kubernetes RBAC (all in-cluster roles, bindings, and authorization — including when using OIDC), application secrets, CCR repository creation, and image push/pull.
### Observability
Crusoe surfaces cluster and hardware metrics to help you monitor performance and utilization. If you install the Crusoe Watch Agent, Crusoe also collects and manages a subset of node-level logs. You are responsible for application-level monitoring and log aggregation.
**Crusoe manages:** Cluster metrics (Prometheus-compatible endpoint), GPU / interconnect / host telemetry, active hardware health checks (AutoClusters), and node-level log collection when the Crusoe Watch Agent is installed.
**You manage:** Application-level monitoring (Grafana, etc.) and log aggregation (Loki, Fluentd, etc.).
### Workloads and Applications
Everything you deploy in the cluster is yours.
**You manage:** All workloads and application-level resources, including Deployments, StatefulSets, DaemonSets, Jobs, namespaces, resource quotas, pod scheduling and affinity rules, custom schedulers, AI/ML training job orchestration (Kubeflow, PyTorchJob, etc.), and ConfigMaps.
## Support Scope
**Supported:** Crusoe owns it. Covered by our service-level commitments. File a ticket; we own resolution. Includes control plane, worker node OS, GPU drivers, Cilium, Persistent Disk infrastructure, cluster metrics.
**Best-Effort:** We investigate and advise, but do not own the outcome. Not covered by SLA. Includes GPU driver issues under specific workload patterns, NCCL tuning, performance optimization, custom scheduling conflicts.
**Customer-Owned:** You install and operate it. If it breaks the cluster, we restore cluster health but will not debug the component. Including but not limited to: service mesh (Istio, Linkerd), custom ingress controllers, custom schedulers, third-party operators, application monitoring, CI/CD tooling, custom admission webhooks.
## Cluster Stability and Component Conflicts
Because your CMK clusters are dedicated to you, you have full control over the workloads and operators you deploy. However, Crusoe remains responsible for the health and uptime of the Control Plane.
If a customer-installed component (such as a custom webhook or third-party operator) causes the Control Plane to fail or degrades the underlying infrastructure, our priority is to restore baseline health. We will notify you to fix or remove the component.
---
_For questions about specific components not covered here, [contact Crusoe support](https://docs.crusoecloud.com/resources/support)._
---
# Cluster Details
This page provides information on key aspects of CMK, such as our current supported versions, cluster components and other operational details.
## Version Support
CMK currently supports Kubernetes version `1.35`, `1.34`, `1.33`, and `1.32`, with planned support for `1.36`. Specific CMK versions are appended with a `-cmk.x` suffix, where x denotes a monotonically increasing count starting from `0`, denoting Crusoe-specific patch version releases.
### Deprecation Calendar
CMK version deprecation dates are typically within 60 days of upstream [End of Life](https://kubernetes.io/releases/) dates, but versions listed below are still officially supported.
| Version | Upstream End of Life Date |
| ------- | ------------------------- |
| `1.32` | March 27th, 2026 |
| `1.33` | June 28th, 2026 |
| `1.34` | October 27th, 2026 |
| `1.35` | February 28th, 2027 |
## Cluster Components
CMK's cluster distribution closely follows upstream Kubernetes. We aim for our distribution to be as 'standard' as possible to simplify installation and configuration of packages and libraries typically used when building AI workloads.
| Component | Details |
| ----------------------------------- | ------------------------------------------------ |
| `Container Runtime` | [containerd](https://containerd.io/) |
| `Cluster DNS` | [CoreDNS](https://coredns.io/) |
| `Container Network Interface (CNI)` | [Cilium](https://cilium.io/) |
| `Worker Node OS` | Ubuntu 22.04 |
| `GPU / Network Driver Installation` | Via operators, available through cluster add-ons |
## Firewall Rules and Secrets
When creating a cluster in a subnet, we create and manage the following firewall rules to manage and maintain connectivity to the cluster. These rules will be created on cluster creation, updated as necessary and removed when clusters are deleted.
Note that for [non-default VPCs](/networking/vpc-networks/managing-vpc-networks), you will need to enable intra-VPC communication between nodes in your subnet for your cluster to successfully bootstrap. We do not create these rules by default. Additionally, the 'Destination Resource' for rules may not be visible via our interfaces given that our control plane nodes are fully managed.
| Component | Details |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cmk-cp-api-access-cp-*` | Provides public API access to the cluster control plane |
| `cmk-cp-core-traffic-cp-*` | Opens up ports 2379, 2380, and 6443 to enable essential communication between control plane components |
| `cmk-cp-metrics-access-cp-*` | Opens up port 24224 on cluster control plane nodes to export metrics |
| `cmk-cp-fluent-bit-access-cp-*` | Opens up port 24224 on cluster control plane nodes to export logs |
| `cmk-cp-konnectivity-cp-*` | Opens up ports 8132–8134 on cluster control plane nodes for Konnectivity server endpoints. Konnectivity is required to enable communication between control plane nodes and worker nodes |
| `cmk-cp-ssh-access-cp-*` | Provides SSH access to control plane nodes from a restricted set of Crusoe-managed IPs |
In addition to firewall rules, we also create an API token with the name `cmk-{clusterName}` that provides API access to relevant cluster components and [add-ons](../cmk/cmk-addons.md) like the Crusoe CSI or Cloud Controller Manager.
## Cluster Upgrades
You may request an upgrade by reaching out to [Crusoe Cloud support](https://support.crusoecloud.com/hc/en-us). Our team will work with you to schedule an appropriate window to upgrade your cluster control planes.
## Cluster Access Control
CMK cluster identity is currently decoupled from Crusoe identity and roles. Upon cluster provisioning, 'admin' users in your organization will be able to retrieve a cluster Kubeconfig tied to an admin role within the cluster. This may be done via the UI, CLI or API.
## Cluster Trust
Communication between control plane and worker nodes, control plane and etcd along with etcd to etcd communication is encrypted via mTLS. Secrets stored in etcd are also [encrypted at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#providers).
---
# Managing your Clusters
# Manage your CMK Clusters
## Creating a New Cluster
**CLI:**
You can create clusters using the `kubernetes clusters create` command. Use the `-help` flag for an exhaustive list of options.
```sh
crusoe kubernetes clusters create \
--name my-first-cluster \
--cluster-version 1.31 \
--location us-east1-a \
--subnet-id 6f8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c
--add-ons "nvidia_gpu_operator,nvidia_network_operator,crusoe_csi"
```
You may list the Kubernetes versions available for cluster creation by using the 'kubernetes clusters list-versions' command. Specifying an unqualified version (e.g. 1.30) when creating a cluster will provision the latest stable patch version associated with the minor version.
**UI:**
To create a CMK cluster via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Click **Create Cluster**.
3. Follow the UI flow to input all required elements.
4. Optional selections include specifying the Service and Pod network CIDRs for Cilium and selecting one or more add-ons to deploy into the cluster.
5. Click **Create**.
**Terraform:**
You can use the `crusoe_kubernetes_cluster` resource to create a new Kubernetes cluster using Terraform.
```hcl
terraform {
required_providers {
crusoe = {
source = "crusoecloud/crusoe"
}
}
}
locals {
my_ssh_key = file("~/.ssh/id_ed25519.pub") # replace with path to your public SSH key if different
}
resource "crusoe_kubernetes_cluster" "my_first_cluster" {
name = "tf-cluster"
version = "1.31.7-cmk.x" # replace with the version you want
location = "us-east1-a"
subnet_id = "6f8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c"
add_ons = ["nvidia_gpu_operator","nvidia_network_operator","crusoe_csi"]
}
```
## Viewing Existing Clusters
**CLI:**
Use the `kubernetes clusters list` command to list all existing clusters. You can also use the `kubernetes clusters get' command to retrieve details for a specific cluster.
```sh
crusoe kubernetes clusters get
```
**UI:**
To view a list of clusters via the Crusoe Cloud console:
```
- Visit the Crusoe Cloud console
- Click the "Orchestration" tab in the left nav
- Select the cluster you want credentials for
- Click the "Generate Kubeconfig" button on the top right
- Your kubeconfig will be downloaded
```
## Get cluster credentials
**CLI:**
Use the `kubernetes clusters get-credentials` command to retrieve credentials for a specific cluster.
```sh
crusoe kubernetes clusters get-credentials
```
By default, credentials are stored in a file named `~/.kube/config`. You may alter the path credentials are stored at by using the `--kubeconfig-path` flag. If you have existing configs stored in the same path, the new cluster kubeconfig will be appended to the end and set as the current context.
**UI:**
If you are an admin user, you can retrieve your cluster admin kubeconfig via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Select the cluster you want credentials for.
3. Click **Generate Kubeconfig** in the top right.
4. Your kubeconfig will be downloaded.
## Delete a cluster
Note that you must delete all nodepools in the cluster before deleting the cluster.
**CLI:**
You can delete clusters by using the `kubernetes clusters delete` command and specifying either the name of ID of the cluster you want to delete.
```sh
crusoe kubernetes clusters delete
```
**UI:**
To delete a CMK cluster via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Select the cluster you want to delete.
3. Make sure that the cluster has no nodepools associated with it. If it does, delete them.
4. Click the **Delete** icon associated with the cluster.
**Terraform:**
Running a `terraform destroy` command provided by the Terraform CLI tool will delete the cluster and associated resources.
If you are having issues creating or deleting clusters, please [contact support](../../resources/support.md).
---
# CMK Add-ons
## Overview
We support a growing set of add-ons and plugins that extend the functionality of CMK (and Kubernetes in general). These add-ons either help integrate CMK with native aspects of Crusoe Cloud, like storage or node lifecycle activities or add features relevant to AI workloads.
We support two methods of add-on installation:
1. When provisioning a cluster, you can opt-in to installing a number of these add-ons
2. At any point post provisioning a cluster, each add-on may be installed via [Helm](https://helm.sh).
Most add-ons that interface with Crusoe Cloud APIs (like the Cloud Controller Manager and Container Storage Interface) require an [API access / secret key](../../identity-and-security/managing-api-keys.mdx) to be present on the cluster and named `CRUSOE_ACCESS_KEY` and `CRUSOE_SECRET_KEY`. By default, we create a managed secret for you during cluster provisioning titled `cmk-{clusterName}`, that is compatible with all Crusoe-vended add-ons.
## Supported Add-ons
| Addon | Details | Links |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Container Storage Interface (CSI)` | The Crusoe CSI allows workloads within the cluster to create and manage supported [Crusoe disk types](../../storage/disks/overview.md) as PersistentVolumes. This currently includes our Persistent Disks and Shared Disks products. | [Github](https://github.com/crusoecloud/crusoe-csi-driver-helm-charts) |
| `Nvidia GPU Operator` | Discovers and exposes Nvidia GPUs as allocatable resources associated with nodes. Currently supported as an opt-in add-on when provisioning a cluster, and may be installed by following the [default configuration](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#procedure) instructions in the documentation. | [Nvidia Docs](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#procedure) |
| `Nvidia Network Operator` | Discovers and exposes Nvidia Host Channel Adapters (HCAs) as allocatable resources within nodes. Currently supported as an opt-in add-on when provisioning a cluster, and is applicable when attaching InfiniBand enabled instances as nodes to your cluster. You may also install the Network Operator by following the [Vanilla Kubernetes](https://docs.nvidia.com/networking/display/cokan10/network+operator#src-39285883_NetworkOperator-NetworkOperatorDeploymentonVanillaKubernetesCluster) installation instructions in the documentation. | [Nvidia docs](https://docs.nvidia.com/networking/display/cokan10/) |
| `Cluster Autoscaler` | Deploys a CMK-compatible [cluster autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) into your cluster that will automatically scale your node pools in and out based on the number of pending pods in your cluster. Please note that autoscaler does not set default min-max limits for node pools. Instructions to configure these limits both during provisioning, and when new node pools are added to the cluster, are available at the linked Github repository. | [Github](https://github.com/crusoecloud/k8s-autoscaler/tree/crusoe-cluster-autoscaler-release-1.30.3/cluster-autoscaler/cloudprovider/crusoecloud) |
| `AutoClusters` | Enhances workload resilience by automatically detecting and remediating node-level hardware failures. AutoClusters gracefully terminates affected pods, restarts or replaces the unhealthy node, and reschedules pods on the healthy replacement, minimizing downtime. See the documentation to learn how to enable AutoClusters for your workloads. | [AutoClusters Docs](./autoclusters.md) |
| `Active Health Checks` | Automated GPU health testing that runs hardware checks on idle nodes to detect issues before they affect your workloads. Tests exercise GPU memory, compute, and interconnect bandwidth at very low priority, ensuring they never interfere with your jobs. Currently in limited availability — contact support to enable. | [Active Health Checks Docs](./active-stress-testing.md), [Github](https://github.com/crusoecloud/autoclusters-controller-helm-charts) |
| `Crusoe Slurm Operator (CSO)` | Deploys a fully managed Slurm cluster on top of CMK, providing GPU job scheduling, multi-user access, shared storage, and topology-aware placement. The operator manages the full Slurm lifecycle including the control plane, login nodes, and worker node sets. AutoClusters integration automatically remediates hardware failures and requeues affected jobs. | [Managed Slurm Docs](../slurm/advanced-kubernetes.md) |
## CSI NFS Support
Starting with CSI driver version **v0.10.6**, the Crusoe CSI driver mounts shared volumes using NFS (Network File System), powered by VAST NFS. This provides improved performance and reliability for shared storage workloads.
### Prerequisites
- Cluster must be on CSI Helm chart **v0.10.6** or newer
- Worker nodes should use NFS-baked worker images (recommended for faster initialization)
### Recommended Images
For optimal NFS support, use the following image versions or later:
#### Control Plane Images (includes CSI v0.10.6+)
| Kubernetes Version | Image |
| ------------------ | --------------- |
| 1.33 | `1.33.4-cmk.43` |
| 1.32 | `1.32.7-cmk.28` |
| 1.31 | `1.31.7-cmk.30` |
#### Worker Images (NFS pre-installed)
| Kubernetes Version | Image |
| ------------------ | --------------- |
| 1.33 | `1.33.4-cmk.4` |
| 1.32 | `1.32.7-cmk.16` |
| 1.31 | `1.31.7-cmk.9` |
| 1.30 | `1.30.8-cmk.16` |
#### GB200 Worker Images (NFS pre-installed)
| Kubernetes Version | Image |
| ------------------ | -------------------- |
| 1.33 | `1.33.4-cmk.5-gb200` |
### Instance Type Compatibility
Shared filesystem volumes with NFS support are available on all instance types:
| Instance Family | Supported Types |
| --------------- | --------------- |
| `c1a` | All types |
| `s1a` | All types |
| GPU instances | All types |
### Migrating from VirtioFS to NFS
Installing CSI Helm chart v0.10.6+ and enabling NFS for your project does **not** automatically convert existing VirtioFS-mounted shared disks to NFS. Existing mounts continue using VirtioFS until they are detached and re-attached.
Once the upgraded CSI Helm chart is installed and NFS is enabled:
- **New shared disks** will mount via NFS
- **New attachments of existing shared disks** will mount via NFS
- **Existing mounts** remain on VirtioFS until re-attached
:::note
On worker nodes without pre-installed NFS packages, the initial NFS package download takes approximately 6 minutes. Using the recommended NFS-baked worker images reduces this to negligible time.
:::
### Migration Approaches
#### Active Migration (Recommended)
In this approach, you intentionally trigger the CSI driver to detach and re-attach shared disks to re-mount them via NFS.
**Important:** The CSI driver uses reference counting to determine when it's safe to unmount/detach a disk. If any pods on a worker node are using a shared disk, the CSI driver will not unmount or detach that disk.
**Option 1: Remove pods to trigger re-attachment**
1. Remove all pods using a specific shared disk from a node using one of these methods:
- Pod affinity/anti-affinity rules
- `topologySpreadConstraints`
- Node draining:
```sh
kubectl drain --ignore-daemonsets --delete-emptydir-data
```
2. Once all pods are removed, the CSI driver will unmount and detach the disk
3. When pods are rescheduled, the disk will re-attach using NFS
**Option 2: Replace node pools**
Create new node pools with NFS-baked worker images and delete the old node pools. This quickly triggers new NFS attachments for all shared disks:
1. Create a new node pool using recommended worker images
2. Cordon old nodes:
```sh
kubectl cordon
```
3. Drain workloads to new nodes:
```sh
kubectl drain --ignore-daemonsets --delete-emptydir-data
```
4. Delete the old node pool once migration is complete
#### Passive Migration
This approach relies on natural churn within your cluster:
- If using the **Cluster Autoscaler** add-on with node pools that frequently scale up and down, nodes will naturally cycle out over time
- As nodes are replaced, new attachments will use NFS
**Considerations:**
- Some long-lived nodes may never cycle out naturally
- You can check node lifetimes with:
```sh
kubectl get nodes
```
- For nodes with long lifetimes, consider manually terminating the underlying Crusoe VM to force a replacement
### Important Notes
- Existing volumes managed by the CSI driver persist through upgrades
- Already-mounted volumes remain unaffected during the driver update
- If upgrading from a version prior to v0.7.0, you must follow the [v0.7.0 upgrade instructions](https://github.com/crusoecloud/crusoe-csi-driver-helm-charts/blob/release/CHANGELOG.md#upgrade-caveats-3) first
:::tip
For assistance with NFS migration or to enable NFS support for your project, contact [Crusoe Cloud Support](https://support.crusoecloud.com/hc/en-us).
---
# Managing your Node Pools
# Manage your Node Pools
## What to know about Node Pools
- Node pools allow you to group one or more Crusoe Cloud instances of the same type and associate them to a specific cluster control plane, to function as worker nodes.
- When creating a node pool, you must specify the number of VMs you want to create via a `count` field. Once specified, the node pool will maintain this VM count where possible. For example, if you stop or terminate one of the VMs in your node pool, the node pool will provision new VMs up till the `count` specified.
- You may scale up or down your node pool by specifying a new `count` value. Note that setting a `count` value lower than the current number does not automatically delete VMs from your node pools. You must manually delete the instances you want removed.
- We currently do not allow editing the startup script or image associated with node pools. To install packages or software on node bring-up, we recommend using [Daemonsets](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/).
## Creating a New Node Pool
**CLI:**
Nodepools can be created by using the `kubernetes nodepools create` command. Nodepools must be created in the context of a specific cluster. Use the '--help' flag for an exhaustive list of options.
```sh
crusoe kubernetes nodepools create \
--name my-first-nodepool \
--cluster-id 6f8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c \
--type h100-80gb-sxm-ib.8x \
--count 4 \
--ib-partition-id 4c8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c \
```
**UI:**
To create a node pool via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Select the cluster you want to edit.
3. Click **Create Node Pool**.
4. Fill out the required fields specifying the type of nodes you want to create and the count.
5. Click **Create**.
**Terraform:**
You can use the `crusoe_kubernetes_node_pool` resource to create a new node pools in the context of a specific cluster via Terraform.
```hcl
terraform {
required_providers {
crusoe = {
source = "crusoecloud/crusoe"
}
}
}
locals {
my_ssh_key = file("~/.ssh/id_ed25519.pub") # replace with path to your public SSH key if different
}
resource "crusoe_kubernetes_cluster" "my_first_cluster" {
name = "tf-cluster"
version = "1.31.7-cmk.x" # Replace with the version you want
location = "us-east1-a"
subnet_id = "6f8e2a1b-7b1d-4c8e-a9f2-8e3d6c1f2a0c"
add_ons = ["nvidia_gpu_operator","nvidia_network_operator","crusoe_csi"]
}
resource "crusoe_kubernetes_node_pool" "l40s_nodepool" {
name = "tf-l40s-nodepool"
cluster_id = crusoe_kubernetes_cluster.my_first_cluster.id
instance_count = "4"
type = "l40s-48gb.10x"
ssh_key = local.my_ssh_key
version = crusoe_kubernetes_cluster.my_first_cluster.version
requested_node_labels = {
# Optional: Kubernetes Node objects will be labeled with the following key:value pairs
# "labelkey" = "labelvalue"
}
}
```
## Viewing Existing Node Pools
**CLI:**
Use the `kubernetes nodepools list` command to list all existing node pools across clusters. You can also use the `kubernetes nodepools get' command to retrieve individual node pool details.
```sh
crusoe kubernetes nodepools get
```
**UI:**
To view your node pools via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Select a cluster.
3. You will see a list of node pools in the **Node Pools** section of the cluster details view.
## Update your Node Pool Template
You can edit the following properties of your node pool:
- Change the count of nodes in the node pool
- Set the pool to use (or not use) local ephemeral NVMe for containerd storage
- Set the labels for your pool. The newly provided labels will overwrite all old labels when new nodes come up.
- Set the taints for your pool. The newly provided taints will overwrite all old taints when new nodes come up.
- The version of the pool's Kubernetes worker nodes. You can get a list of available versions by running `crusoe kubernetes versions list` from the Crusoe CLI.
Updating properties of a nodepool will update the nodepool template, but will not perform an in-place upgrade of the existing nodes in the pool. If you scale the nodepool after applying an update, the updated template will only apply to the newly created nodes. To update existing nodes, you must perform a rolling upgrade (see section below).
**CLI:**
You may update the number of nodes in your nodepools by using the `crusoe kubernetes nodepools update` command.
```sh
crusoe kubernetes nodepools update --count 3
```
For instructions on how to update additional elements of your nodepool (e.g. ephemeral storage for containerd, nodepool labels), run:
```sh
crusoe kubernetes nodepools update -h
```
**UI:**
To update the number of VMs in your node pool via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Select the cluster you want to update.
3. Select the edit icon next to the node pool you want to update.
4. Specify the new number of nodes that you want to add.
5. Click **Update**.
**Terraform:**
```hcl
resource "crusoe_kubernetes_node_pool" "l40s_nodepool" {
name = "tf-l40s-nodepool"
cluster_id = crusoe_kubernetes_cluster.my_first_cluster.id
# Optional: Set the desired instance count
instance_count = "6" # add 2 more nodes
# Optional: Set the desired CMK worker node version
# If not specified, the default is the latest stable version compatible with the cluster
# List available node pool versions with "crusoe kubernetes versions list"
version = "1.31.7-cmk.x" # Replace with the version you want
type = "l40s-48gb.10x"
# Optional: Kubernetes Node objects will be labeled with the following key:value pairs
# requested_node_labels = {
# "labelkey" = "labelvalue"
# }
# Optional: Use local ephemeral NVMe disks for containerd storage
# ephemeral_storage_for_containerd = true
}
```
## Updating Nodes in Your Node Pool
To make existing nodes reflect the latest nodepool configuration template, simply delete the VMs backing the nodes. The node pool will heal by creating new VMs that reflect the new template.
## Delete a Node Pool
**CLI:**
You can delete nodepools by using the `kubernetes nodepools delete` command and specifying either the name of ID of the nodepool you want to delete.
```sh
crusoe kubernetes nodepools delete
```
**UI:**
To delete a node pool via the [console](https://console.crusoecloud.com):
1. From the console, select **Orchestration** > **[Kubernetes](https://console.crusoecloud.com/orchestration/kubernetes)** in the left nav.
2. Select the cluster you want to update.
3. Select the delete icon next to the node pool you want to delete.
**Terraform:**
Deleting nodepools may be accomplished by removing the desired `crusoe_kubernetes_nodepool` resource from your terraform configuration.
If you are having issues creating or deleting clusters, please [contact support](../../resources/support.md).
---
# CMK Telemetry
Command Center gives you visibility into the health, performance, and behaviors of your Crusoe Managed Kubernetes Clusters. Three categories of telemetry are available:
**Metrics:** Infrastructure metrics covering GPU (via DCGM), CPU, memory, disk, network, and InfiniBand performance, are collected at 60-second intervals and retained for 30 days. Custom application metrics are also supported: expose metrics in Prometheus format from your pods and annotate them for scraping alongside infrastructure metrics. A subset of metrics is viewable in the Console. The full dataset is available via Prometheus-compatible API, Grafana, and Telemetry Conduit.
From the [Console](https://console.crusoecloud.com/), navigate to **Orchestration**, select your cluster, and then select the **Metrics** tab to see aggregated cluster-level views. Select a node to drill into node-level detail.
**Logs:** JournalD system logs, kubelet logs, container runtime logs, and container logs are collected from each node and are available to search, filter, and query. Logs are retained for 7 days. In the Console, navigate to **Orchestration**, select your cluster, then select the **Logs** tab. Logs are also accessible via LogsQL API.
**Bug Reports:** Generate an NVIDIA or AMD bug report for any node from the Console, the CLI, or the API. NVIDIA bug reports include nvidia-smi output and kernel XID logs. In the Console, navigate to **Orchestration**, select your cluster, select a node, then use the action menu to generate a report. For requirements, collection steps, and error messages, see [Diagnostics](../../command-center/diagnostics.mdx).
For installation, token generation, and access method details, see [Get started](../../command-center/get-started.mdx).
Pre-built Grafana dashboard templates for CMK clusters are available in the [Crusoe solutions library](https://github.com/crusoecloud/solutions-library/tree/main/grafana-cmk). Templates cover GPU utilization, InfiniBand fabric health, power draw, XID error tracking, storage, and network.
## Available Metrics
The following metrics are available at the cluster level in the Crusoe Console and via API. Node-level metrics are available in the VM telemetry view. Visit [VM Telemetry](../../compute/virtual-machines/vm-telemetry.md) for the full list of node-level metrics and their query parameters.
| **Metric** | **Definition** | **Suggested Query** |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cluster TFLOPS (FP16) | Average 16-bit GPU throughput (TFLOPS) across a node pool, based on scaled Tensor Core utilization. | `avg by (nodepool) (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{cluster_id="${clusterId}"}[60s])` \* theoretical max TFLOPS |
| Average GPU Utilization (%) | The GPU utilization, averaged across all GPUs within a node pool. | `avg by (nodepool) (DCGM_FI_DEV_GPU_UTIL{cluster_id="${clusterId}"}[60s])` |
| Cumulative Power Usage (kW) | Cumulative power usage summed up across all nodes in a node pool. | `sum by (nodepool) (DCGM_FI_DEV_POWER_USAGE{cluster_id="${clusterId}"}[60s]) / 1000` |
| Average GPU Memory Utilization (%) | The GPU memory utilization, averaged across all GPUs within a node pool. | `avg by (nodepool) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{cluster_id="${clusterId}"}[60s]))` |
| Average CPU Utilization (%) | The CPU utilization, averaged across all CPU cores within a node pool. | `sum by (nodepool) (rate(crusoe_vm_cpu_seconds_total{cluster_id="${clusterId}", mode!="idle"}[60s])))/(sum by (nodepool) (rate(crusoe_vm_cpu_seconds_total{cluster_id="${clusterId}"}[60s])) * 100` |
| Aggregated VPC Network Bandwidth In (bytes per second) | The rate of data received via the VPC network interface, aggregated across all nodes in a node pool. | `sum by (nodepool) (rate(crusoe_vm_network_receive_bytes_total{cluster_id="${clusterId}", device!~"lo"}[60s]))` |
| Aggregated VPC Network Bandwidth Out (bytes per second) | The rate of data transmitted via the VPC network interface, aggregated across all nodes in a node pool. | `sum by (nodepool) (rate(crusoe_vm_network_transmit_bytes_total{cluster_id="${clusterId}", device!~"lo"}[60s]))` |
| Uncorrectable ECC Error Rate | The rate of uncorrectable double-bit memory errors (DBE) accumulated across all GPUs in a node pool. | `sum by (nodepool) (rate(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL{cluster_id="${clusterId}"}[RANGE]))` |
| Correctable ECC Error Rate | The rate of correctable single-bit memory errors (SBE) accumulated across all GPUs in a node pool. | `sum by (nodepool) (rate(DCGM_FI_DEV_ECC_SBE_VOL_TOTAL{cluster_id="${clusterId}"}[60s]))` |
| InfiniBand Throughput Rx (bytes per second) | The rate of data received via InfiniBand port, aggregated across all nodes in a cluster. | `sum by(cluster_id) (rate(crusoe_ib_port_throughput_rx{cluster_id="${clusterId}"}[5m]) / 1000)` |
| InfiniBand Throughput Tx (bytes per second) | The rate of data transmitted via InfiniBand port, aggregated across all nodes in a cluster. | `sum by(cluster_id) (rate(crusoe_ib_port_throughput_tx{cluster_id="${clusterId}"}[5m]) / 1000)` |
XID error logs with error details are also available in **Orchestration** > cluster > **Metrics** tab.
## Custom Metrics
You can ingest custom application metrics alongside infrastructure metrics for end-to-end visibility from hardware to application performance. Custom metrics are available for CMK clusters only.
To expose custom metrics, format them in Prometheus format on an HTTP endpoint and annotate your pods to enable scraping:
```yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
crusoe.ai/scrape: "true"
crusoe.ai/port: "8080"
crusoe.ai/path: "/my-app/metrics"
spec:
containers:
- name: my-training-job
image: my-training-image:latest
ports:
- containerPort: 8080
```
Custom metrics are available through the same API endpoint as infrastructure metrics and can be queried using PromQL.
:::note
Custom metrics aren't available in the Console. Use the API or Grafana to query custom metrics.
:::
For token generation and querying metrics via API or Grafana, see [Get started](../../command-center/get-started.mdx) and [Metrics](../../command-center/metrics.md).
## Considerations
### Parsing errors caused by special characters
To prevent parsing errors caused by special characters like `$` in the monitoring token during Helm chart deployment, reference the token from a Kubernetes Secret using the `secretKeyRef` mechanism:
```yaml
# In your application's Deployment or Pod manifest
env:
- name: CRUSOE_MONITORING_TOKEN
valueFrom:
secretKeyRef:
name: crusoe-monitoring-token # must match the name of your Kubernetes Secret
key: CRUSOE_MONITORING_TOKEN # must match the key used inside the Secret object
```
---
# AutoClusters
# Automated Node Remediation with AutoClusters
AutoClusters enhances the resilience of your CMK workloads by automatically detecting and resolving common hardware failures. By enabling AutoClusters you can minimize downtime and reduce the need for manual intervention, ensuring higher effective utilization for your clusters.
This guide explains how AutoClusters works, how to enable it for your deployments, and what to expect during the automated remediation process.
## Supported Versions and Hardware
AutoClusters is currently supported on the following minimum CMK versions:
- `1.33.4-cmk.20`
- `1.32.7-cmk.22`
- `1.31.7-cmk.24`
Kubernetes minor versions greater than `1.33` are all supported, for all patch versions. Future minor and patch releases within the above Kubernetes versions (e.g., `1.33.5-cmk.X`, `1.32.8-cmk.X`) are also supported.
AutoClusters can remediate issues for Kubernetes nodes running on the following Crusoe GPU instance types:
- 10x NVIDIA L40S (`l40s-48gb.10x`)
- 8x Nvidia A100 80GB (`a100-80gb.8x`, `a100-80gb-sxm-ib.8x`)
- 8x Nvidia H100 80GB (`h100-80gb-sxm-ib.8x`)
- 8x NVIDIA H200 141GB (`h200-141gb-sxm-ib.8x`)
- 8x NVIDIA B200 180GB (`b200-180gb-sxm-ib-8x`)
- 4x NVIDIA GB200 186GB (`gb200-186gb-nvl-4x`)
- 8x NVIDIA B300 288GB (`b300-288gb-sxm-ib.8x`)
AutoClusters does **not** support remediation on multi-tenant VM types where multiple Kubernetes nodes may be co-located on the same physical host. In those configurations, node replacement cannot be safely and deterministically executed, and AutoClusters will not trigger remediation.
## How it Works
AutoClusters continuously monitors your infrastructure for hardware-related errors. When a critical issue is detected on a node, AutoClusters initiates a remediation process based on standard Kubernetes procedures.
The process involves:
1. **Graceful Termination:** Your workloads are given time to shut down cleanly.
2. **Node Restart or Replacement:** The unhealthy node is either restarted or removed from the cluster and replaced with a healthy one.
3. **Workload Rescheduling:** Your pods are automatically rescheduled onto the new node.
This entire process is automated, allowing your workloads to recover from hardware failures without manual intervention.
Remediation only runs for the issue types you have turned on. When the AutoClusters addon is first enabled, every issue type defaults to `OFF`, so AutoClusters detects failures and sends notifications but does not replace any nodes until you enable remediations. See [Enable Remediations](#step-2-enable-remediations).
## Enabling AutoClusters
### Step 1: Enable the AutoClusters Addon
AutoClusters is a Kubernetes add-on that must be enabled at the cluster level.
- **For new clusters:** You can enable the AutoClusters addon during the cluster creation process through the UI by selecting the AutoClusters add-on, or through the CLI via the `--add-ons` flag.
- **For existing clusters:** Please contact our support team to have the AutoClusters addon enabled for your existing cluster.
:::info
Enabling the addon turns on hardware-failure **detection**, but **all remediation actions default to `OFF`**. No nodes are replaced until you explicitly enable remediations in [Step 2](#step-2-enable-remediations). Until then AutoClusters will take no automatic action.
:::
### Step 2: Enable Remediations
Because every issue type defaults to `OFF`, you turn AutoClusters on by setting the issue types you want it to act on to `REPLACE_NODE`. This gives you control over exactly which hardware failures trigger automatic node replacement.
**First, review your current configuration.** This is especially important if AutoClusters was set up on this cluster previously — you may already have overrides in place, and checking first avoids unexpected changes:
```sh
crusoe kubernetes autoclusters config get --project-id YOUR_PROJECT_ID --cluster-id YOUR_CLUSTER_ID
```
This lists every supported issue type with its default action, any override you've set, and the resulting effective action. See [Remediation Configuration](#remediation-configuration) for how to read this output.
**Then enable remediations.** To turn on automatic node replacement for all supported issue types in a single command:
```sh
crusoe kubernetes autoclusters config set-remediation-override \
--project-id YOUR_PROJECT_ID \
--cluster-id YOUR_CLUSTER_ID \
--override GPU_FELL_OFF_THE_BUS=REPLACE_NODE \
--override HCA_FELL_OFF_THE_BUS=REPLACE_NODE \
--override HCA_POLLING=REPLACE_NODE \
--override NVSWITCH_FELL_OFF_THE_BUS=REPLACE_NODE \
--override PCI_LINK_DOWN=REPLACE_NODE \
--override XID_119=REPLACE_NODE \
--override XID_120=REPLACE_NODE \
--override XID_48=REPLACE_NODE \
--override XID_64=REPLACE_NODE \
--override XID_74=REPLACE_NODE \
--override XID_79=REPLACE_NODE
```
To enable only certain issue types, include just those `--override` flags. See [Remediation Configuration](#remediation-configuration) for per-issue control and how to turn remediations back off.
:::warning
Once remediations are enabled, AutoClusters will drain and replace nodes when it detects the corresponding hardware failures. Implement `preStop` hooks for any workloads that require graceful shutdown, and use the opt-out label below for any workloads that should never trigger node replacement.
:::
### Opting Out of Automated Remediation (Optional)
Once you have enabled remediations, all nodes are eligible for automated node replacement. If you have specific workloads that should prevent automatic node replacement, you can opt them out by adding the `autoclusters.crusoe.ai/remediationPolicy: "Disabled"` label to the pod template of your resource (e.g., a Job or Deployment).
Here's an example for a Kubernetes Deployment:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: your-deployment-name
spec:
# ...
template:
metadata:
labels:
autoclusters.crusoe.ai/remediationPolicy: "Disabled"
# ...
```
All pods created from this template will inherit the label, preventing automatic remediation on any node where these pods are running. If any pod on a node has this label set to `"Disabled"`, AutoClusters will not remediate that node.
### Implementing `preStop` Hooks for Graceful Shutdown
As a Kubernetes best practice, you should implement a `preStop` hook in your containers to ensure graceful shutdown during pod termination. This is important not just for AutoClusters remediation, but for any scenario where pods are deleted or rescheduled (such as deployments, node maintenance, or resource constraints).
Without a `preStop` hook, when remediation occurs, your workloads will be moved to another node but any in-progress work may be lost. The `preStop` hook is executed before the pod is terminated, giving your application time to save its state, checkpoint its work, or close any open connections.
Here's an example of a `preStop` hook that runs a shell script to save state:
```yaml
spec:
template:
# ...
spec:
containers:
- name: my-container
image: my-image
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "/app/your-save-state-script.sh"]
# Optionally set a grace period for your hook to complete. If you do not set this value, the default grace period of 30 seconds will apply.
terminationGracePeriodSeconds: 120
# ...
```
In this example, `your-save-state-script.sh` is a script you provide to handle the graceful shutdown of your application.
## Remediation Configuration
Remediation behavior is controlled per issue type. Use these commands to inspect the current configuration and to enable or disable automatic node replacement for individual issue types.
### Viewing Current Configuration
To view your cluster's remediation configuration, including both default behavior and any overrides:
```sh
crusoe kubernetes autoclusters config get --project-id YOUR_PROJECT_ID --cluster-id YOUR_CLUSTER_ID
```
This shows all supported issue types (e.g., `XID_64` for GPU errors), their default action, any override you've configured, and the resulting effective action. Because the default action for every issue type is `OFF`, an issue is only remediated automatically if you have set an override of `REPLACE_NODE` for it.
### Setting Remediation Actions
Each issue type has two possible actions:
- **`REPLACE_NODE`**: Automatically drain and replace the node when the issue is detected.
- **`OFF`**: Take no automatic action — you still receive notifications, but the node is not replaced. This is the default for every issue type until you enable remediations.
Setting an action creates an override for that issue type. To enable automatic node replacement for a specific issue type (for example, `XID_64`):
```sh
crusoe kubernetes autoclusters config set-remediation-override \
--project-id YOUR_PROJECT_ID \
--cluster-id YOUR_CLUSTER_ID \
--override XID_64=REPLACE_NODE
```
To turn remediation back off for a specific issue type:
```sh
crusoe kubernetes autoclusters config set-remediation-override \
--project-id YOUR_PROJECT_ID \
--cluster-id YOUR_CLUSTER_ID \
--override XID_64=OFF
```
To remove an override entirely and return the issue type to its default action (for example `XID_64`):
```sh
crusoe kubernetes autoclusters config remove-remediation-override \
--project-id YOUR_PROJECT_ID \
--cluster-id YOUR_CLUSTER_ID \
--issue XID_64
```
To enable every supported issue type at once, see the one-shot command in [Step 2: Enable Remediations](#step-2-enable-remediations).
## Advanced Configuration and Extensibility
:::info Limited Availability Feature
The advanced features described in this section are in Limited Availability. Ask Crusoe Support to enable them for your account.
:::
Beyond enabling and disabling automatic remediation, you may want to implement your own remediation logic or integrate with external systems. AutoClusters extensibility allows you to **trigger remediation manually** via API when your custom logic determines it's appropriate — for example, to build an open-loop workflow that pairs AutoClusters detection with your own decision-making.
These features are currently available through the CLI and API. UI support will be added in future releases. This documentation covers CLI usage. See the [API docs](https://docs.crusoecloud.com/api/#tag/AutoClusters) for API usage.
### Manual Remediation
Manual remediation allows you to trigger node replacement on-demand via API, giving you complete control over when remediation occurs.
#### Eligibility Requirements
You can trigger manual remediation for a VM only when:
1. AutoClusters detected a qualifying issue on the VM within the past 24 hours
2. The VM is on a single-tenancy node
3. The cluster has the AutoClusters add-on enabled
#### Triggering Manual Remediation
```sh
crusoe kubernetes autoclusters remediations replace-node \
--project-id YOUR_PROJECT_ID \
--cluster-id YOUR_CLUSTER_ID \
--vm-id YOUR_VM_ID \
```
Manual remediation events appear in your cluster's AutoClusters tab in the Crusoe Cloud Console.
### Open Loop Workflow
You can implement fully custom remediation logic by combining remediation overrides with webhooks:
1. **Keep automatic remediation off** for the issue types you want to handle yourself. `OFF` is the default, so no action is needed unless you previously enabled the issue type — in which case set it back to `OFF`:
```sh
crusoe kubernetes autoclusters config set-remediation-override \
--project-id YOUR_PROJECT_ID \
--cluster-id YOUR_CLUSTER_ID \
--override XID_64=OFF
```
2. **Configure a webhook** to receive notifications when AutoClusters detects issues. See [Configuring Webhook Notifications](../../notifications/overview.md#configuring-webhook-notifications) for setup instructions.
3. **Implement your custom logic** to decide when and how to remediate. Your webhook endpoint receives issue detection events and can implement any decision logic you need (e.g., checking external systems, waiting for specific conditions, coordinating with schedulers).
4. **Call the remediation API** when your logic determines it's appropriate:
```sh
crusoe kubernetes autoclusters remediations replace-node \
--project-id YOUR_PROJECT_ID \
--cluster-id YOUR_CLUSTER_ID \
--vm-id YOUR_VM_ID \
```
This gives you complete control over the remediation process while still leveraging AutoClusters' detection capabilities and node replacement infrastructure.
## The Remediation Process
When AutoClusters detects a hardware failure that requires node replacement (or when the remediation API is manually triggered), it automatically performs the following steps:
1. **Verification:** AutoClusters confirms that no pods on the node have the `autoclusters.crusoe.ai/remediationPolicy: "Disabled"` label. If any pod on the node has this label set to `"Disabled"`, AutoClusters will not remediate the node.
2. **Node Drain:** The Kubernetes node is cordoned and drained, which gracefully evicts all pods. Your container `preStop` hooks are executed at this stage.
3. **Node Replacement:** The unhealthy node is removed from the node pool and replaced with a new, healthy node from spare capacity. The system first checks to see if there are any suitable spare nodes in the Crusoe on-demand pool. If none are found, the system proceeds to look for suitable spares in Crusoe's hot spare inventory. If no spares can be found, the Crusoe team is notified to take action. You will also be sent a notification via e-mail (or other notification channels you have set up) to take action as necessary.
4. **History Logging:** A record of the alert and the remediation event is logged and made available in the Crusoe Cloud Console for your review.
## Node Remediation History
You can view the history of remediation events for a cluster on the **Remediations** tab in the cluster detail view.
## Edge Cases and Limitations
While AutoClusters is designed to handle failures automatically, there are some important things to be aware of:
- **Remediation Failures:** If the triggered remediation action fails for any reason (e.g. no spare nodes are available to replace the unhealthy one), the remediation process will be aborted, you will be notified, and our support team will be notified to assist.
- **Excessive Remediations:** If a high number of remediation actions are triggered in a short time period, circuit breakers will pause future remediations and our support team will step in to investigate and address underlying causes.
- **Conservative Alert Triggers:** AutoClusters uses a comprehensive but conservative set of hardware failure detection rules to avoid over-remediation. We only remediate for failures we know require node replacement, ensuring your workloads aren't disrupted unnecessarily.
---
# Active Health Checks
Active Health Checks runs automated GPU hardware checks on idle nodes in your CMK cluster. By periodically exercising GPU hardware while nodes are not serving workloads, Active Health Checks detects issues, such as memory errors and interconnect degradation, before they impact your jobs. This helps maintain the health and reliability of your GPU infrastructure at scale.
## Enabling Active Health Checks
Active Health Checks is an opt-in add-on that can be selected during cluster creation.
These are the minimum Kubernetes cluster version numbers that are required to create a cluster with Active Health Checks
- `1.33.4-cmk.50`
- `1.32.7-cmk.35`
- `1.31.7-cmk.37`
- `1.34.2-cmk.6`
:::info Limited Availability
Active Health Checks is currently in limited availability. To enable it for your account, contact [Crusoe Cloud Support](https://support.crusoecloud.com/hc/en-us).
:::
- **New clusters:** Select the Active Health Checks add-on during cluster creation through the UI, or via the `--add-ons` flag in the CLI.
- **Existing clusters:** Contact [Crusoe Cloud Support](https://support.crusoecloud.com/hc/en-us) to have Active Health Checks enabled on your cluster.
## How It Works
When Active Health Checks is enabled, automated test workloads are scheduled periodically on idle nodes in your cluster. Active Health Checks is supported on all GPU instance types (with the exception of B200, due to a known bug awaiting resolution). These tests exercise key aspects of GPU hardware, including:
- **GPU memory and compute** — verifies that GPU memory and core compute operations are functioning correctly.
- **Interconnect bandwidth** — on NVLink-equipped instances, measures GPU-to-GPU communication bandwidth to ensure interconnect performance meets expected baselines.
Test workloads run with the lowest Kubernetes priority. This means they are always the first to be evicted when a real workload needs resources — the Kubernetes scheduler will immediately preempt any running test to make room for your jobs.
## Impact on Your Workloads
Active Health Checks is designed to have **zero impact** on your workloads:
- Tests **only run on idle nodes** and do not consume GPU resources alongside your workloads.
- If you schedule a workload on a node where a test is running, the test is **immediately preempted** by the Kubernetes scheduler. Your workload takes priority.
- **No action is required** from you. Tests run automatically in the background.
## Identifying Test Workloads
If you inspect your cluster and notice unfamiliar workloads, Active Health Checks workloads can be identified by the following:
- They run in the **`crusoe-system`** namespace.
- They run at the lowest Kubernetes priority.
- They are labeled with **`app: autoclusters`**.
## Custom Schedulers
:::warning
Do **not** enable Active Health Checks if you are running a custom scheduler — such as [Volcano](https://volcano.sh/) or [KAI Scheduler](https://github.com/NVIDIA/KAI-Scheduler) — that does not respect standard Kubernetes PriorityClasses and preemption. Custom schedulers that bypass the default Kubernetes scheduling logic may not correctly preempt test workloads, which could lead to resource conflicts.
If you are unsure whether your scheduler is compatible, contact [Crusoe Cloud Support](https://support.crusoecloud.com/hc/en-us) before enabling Active Health Checks.
:::
## What Happens When a Test Fails
When a node fails an Active Health Checks check, the Crusoe team is automatically notified to investigate the failure. You may be contacted by support if any action is needed on your side.
Additionally, failed nodes are annotated with Kubernetes node conditions that you can use in your scheduling logic:
- **`GPUDCGMUnhealthy`** — set when GPU memory or compute diagnostics fail.
- **`GPUNVBandwidthUnhealthy`** — set when interconnect bandwidth falls below expected thresholds.
You can use these conditions with `nodeAffinity` rules or custom logic to avoid scheduling workloads on nodes that have not yet been investigated.
---
# Support Access
Support access allows Crusoe support engineers to access your CMK cluster for troubleshooting and maintenance purposes. When enabled, authorized Crusoe support personnel can perform diagnostic tasks, investigate issues, and provide technical assistance directly within your cluster.
This feature gives you control over when and how Crusoe support can access your infrastructure, ensuring you maintain visibility and control over cluster access.
## Enabling Support Access
Support access is **disabled by default** on all clusters. You can enable it at any time using the CLI command. For existing clusters, you will also need to install a Helm chart.
### Step 1: Enable Support Access via CLI
Enable support access on any cluster using the CLI:
```sh
crusoe kubernetes clusters support-access enable \
--enabled-roles readonly,operator
```
You can choose which level of access to grant:
- **readonly** - View-only access to cluster resources for diagnostics (default)
- **operator** - Additional permissions for troubleshooting and configuration assistance
- **readonly,operator** - Both roles (recommended for comprehensive support)
Note that operator access will automatically enable readonly; so the last two are equivalent.
To enable only readonly access:
```sh
crusoe kubernetes clusters support-access enable
```
### Step 2: Install the Helm Chart (For Existing Clusters)
If you have an existing cluster, you will also need to install a Helm chart that provides the necessary roles and permissions:
1. Add the Crusoe support roles Helm repository:
```sh
helm repo add crusoe-support https://crusoecloud.github.io/crusoe-support-roles-helm-charts/charts
helm repo update
```
2. Install the support access chart:
```sh
helm install crusoe-support-roles crusoe-support/crusoe-support-roles \
--namespace crusoe-system \
--create-namespace
```
The Helm chart is available at: [https://github.com/crusoecloud/crusoe-support-roles-helm-charts](https://github.com/crusoecloud/crusoe-support-roles-helm-charts)
:::info
The Helm chart is required for existing clusters to enable the necessary cluster roles and role bindings in the `crusoe-system` namespace. New clusters created after this feature was released will have this chart pre-installed and only require the CLI command (Step 1).
:::
:::note
By default, CMK clusters include a firewall rule (`cmk-cp-api-access-cp-`) in the associated project that allows all inbound traffic to port 443 on the control plane nodes. If you have restricted this access, you must also allow inbound connections from Crusoe's internal VPN exit node (**4.7.95.218**) so that support engineers can reach the Kubernetes API server.
:::
## Disabling Support Access
If you need to revoke support access to your cluster, you can do so at any time.
### Disable via CLI
Disable support access using the CLI:
```sh
crusoe kubernetes clusters support-access disable
```
:::info
Disabling via CLI prevents new or renewed support access sessions but does not immediately revoke active access. For immediate revocation, see the Helm chart removal option below.
:::
### Uninstall the Helm Chart (For Immediate Revocation)
You can uninstall the Helm chart to **immediately** disable all support access:
```sh
helm uninstall crusoe-support-roles --namespace crusoe-system
```
:::warning
Removing the Helm chart will immediately revoke support access by removing the necessary cluster roles and role bindings from your cluster. Use this method when you need to ensure support access is terminated right away.
:::
**When to use each method:**
- **CLI disable** - Prevents future access while allowing current support sessions to complete
- **Helm chart removal** - Immediately terminates all support access (for existing clusters with the Helm chart installed)
- **Both** - Use both methods for comprehensive access revocation on existing clusters
:::tip
To remove **all** access—including any node-shell or other diagnostic pods that a support engagement may have left running in the Crusoe-managed namespaces—follow the complete checklist in [Removing All Support Access](./removing-all-access.md).
:::
## Verifying Support Access Status
You can verify whether support access is currently enabled on your cluster:
```sh
crusoe kubernetes clusters support-access get
```
This command displays the current support access configuration, including:
- Whether access is enabled
- Which roles are granted
- **Currently active support access sessions** with details about each session
### Understanding the Output
Example output when support access is enabled with an active session:
```
Support Access Status: Enabled
Enabled Roles: SUPPORT_ACCESS_ROLE_READONLY
Active Requests: 1
Request 1:
Role: SUPPORT_ACCESS_ROLE_READONLY
Requestor ID: 00uix9g0n77KrcdMq5d7
Reason: This is a test!
Requested At: 2026-03-30T23:12:53Z
Request Expires At: 2026-04-01T03:12:53Z
Credential Expires At: 2026-04-01T01:46:18Z
```
**Active Requests** shows currently live support credentials, including:
- **Role** - The access level granted (READONLY or OPERATOR)
- **Requestor ID** - The identifier of the support engineer
- **Reason** - The stated purpose for the access request
- **Requested At** - When the access was granted
- **Request Expires At** - When the access will automatically expire
- **Credential Expires At** - When the latest certificate will expire
This transparency allows you to monitor exactly who has access to your cluster and why. Note that if support access is
disabled to the corresponding role, the requests are no longer valid but already minted certificates will not be invalidated
(because Kubernetes does not support certificate revocation). For immediate revocation, see instructions above.
### Checking Helm Chart Installation
If you need to check whether the Helm chart is already installed on your cluster:
```sh
helm list --namespace crusoe-system
```
## What Access Does Support Have?
The level of access granted depends on which roles you enable:
### Readonly Role
With the `readonly` role, Crusoe support engineers can:
- View cluster resources and configurations
- Access logs and metrics for troubleshooting
- Inspect resource status and health
- Diagnose issues without making changes
### Operator Role
With the `operator` role, Crusoe support engineers have additional capabilities to:
- Execute diagnostic commands
- Assist with cluster configuration
- Perform troubleshooting actions
- Help resolve operational issues
For full details of this access, you can reference the helm chart README at
[https://github.com/crusoecloud/crusoe-support-roles-helm-charts/blob/main/charts/crusoe-support-roles/README.md](https://github.com/crusoecloud/crusoe-support-roles-helm-charts/blob/main/charts/crusoe-support-roles/README.md)
### What Support Cannot Access
Regardless of which roles are enabled, support access does **not** grant permission to:
- Modify or delete your workloads without coordination
- Access data within your application containers
- Make infrastructure changes outside of coordinated support engagements
- Access secrets or sensitive application data
## Getting Help
If you encounter issues enabling or disabling support access, or have questions about what level of access is granted, please [contact support](../../resources/support.md).
---
# Removing All Support Access
This addendum to [Support Access](./support-access.md) covers how to remove **all** Crusoe support access to a cluster, going beyond disabling future sessions. Use this procedure when you want to ensure that no Crusoe personnel retain any path into the cluster.
## When to Use This Procedure
The standard [Disabling Support Access](./support-access.md#disabling-support-access) steps prevent new sessions and—when you uninstall the Helm chart—remove the cluster roles and role bindings that support relies on. However, a support engagement may have left behind workloads that retain access to nodes even after the RBAC roles are gone. Removing all access means revoking RBAC **and** cleaning up any such workloads.
## Step 1: Revoke RBAC Access
Follow both methods described in [Disabling Support Access](./support-access.md#disabling-support-access):
1. Disable support access via the CLI to prevent new or renewed sessions:
```sh
crusoe kubernetes clusters support-access disable
```
2. Uninstall the Helm chart to immediately remove the support cluster roles and role bindings:
```sh
helm uninstall crusoe-support-roles --namespace crusoe-system
```
:::note
Already-minted certificates remain valid until they expire, because Kubernetes does not support certificate revocation. Removing the cluster roles and role bindings (Step 1, item 2) ensures those credentials can no longer authorize any action.
:::
## Step 2: Remove Node-Shell and Similar Pods
During troubleshooting, support engineers may run helper pods—such as `node-shell` pods—that mount host paths or open a shell onto a node. These pods can provide node-level access independent of the RBAC roles you removed in Step 1, so they must be deleted explicitly.
Crusoe support roles are scoped to the following Crusoe-managed namespaces. Sweep each of them for leftover diagnostic pods:
- `crusoe-system`
- `kube-system`
- `nvidia-gpu-operator`
- `nvidia-network-operator`
- `kube-amd-gpu`
- `kube-amd-network`
- `slinky`
- `slurm`
1. Check each Crusoe-managed namespace for any such pods. For example:
```sh
kubectl get pods --namespace crusoe-system
```
To check all of the namespaces at once:
```sh
for ns in crusoe-system kube-system nvidia-gpu-operator nvidia-network-operator kube-amd-gpu kube-amd-network slinky slurm; do
echo "== $ns =="
kubectl get pods --namespace "$ns"
done
```
Look for pods with names like `node-shell-*`, debug/shell pods, or any pod you do not recognize as part of normal cluster operation.
2. Delete any node-shell or similar diagnostic pods you find:
```sh
kubectl delete pod --namespace
```
:::warning
Only delete pods that you have confirmed are diagnostic or support-related. Deleting pods that are part of normal cluster operation may disrupt your workloads.
:::
## Step 3: Verify
Confirm that support access is fully disabled and no active sessions remain:
```sh
crusoe kubernetes clusters support-access get
```
Confirm the Helm chart is no longer installed:
```sh
helm list --namespace crusoe-system
```
Confirm no node-shell or diagnostic pods remain in any of the Crusoe-managed namespaces:
```sh
for ns in crusoe-system kube-system nvidia-gpu-operator nvidia-network-operator kube-amd-gpu kube-amd-network slinky slurm; do
echo "== $ns =="
kubectl get pods --namespace "$ns"
done
```
## Getting Help
If you have questions about fully revoking support access or identifying which pods are safe to remove, please [contact support](../../resources/support.md).
---
# Overview
# Crusoe Managed Slurm
Crusoe Managed Slurm provides managed HPC cluster orchestration on Crusoe Cloud. With a single UI form or CLI command, you can provision a complete Slurm cluster backed by Crusoe's GPU-optimized infrastructure.
Managed Slurm combines Slurm's industry-standard job scheduling with Crusoe Managed Kubernetes (CMK), giving you a production-ready HPC environment with topology-aware scheduling, shared storage, and multi-user access.
You can deploy Managed Slurm in either of two modes — both produce the same end-state cluster, with the same operator, the same CRDs, and the same Slurm experience for end users. Pick the path that fits how you want to manage infrastructure:
| Mode | Best for | How you create the cluster | Where to start |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------- |
| **Slurm CLI / UI mode** | Teams who want a fully managed Slurm experience without managing CMK directly. One command provisions everything. | `crusoe slurm clusters create` (or the **Orchestration > Slurm** view in the Console) | [Quickstart](./quickstart.md) |
| **Kubernetes add-on mode** | Teams who already use CMK, want kubectl/Helm-native control of Slurm components, or want to manage Slurm alongside other Kubernetes workloads. | Create a CMK cluster with the `crusoe_managed_slurm` add-on, then apply CRDs | [Set Up Slurm on Kubernetes](./kubernetes-setup.md) |
## How It Works
When you create a Managed Slurm cluster, Crusoe automatically provisions:
1. **A CMK cluster** with all required add-ons and networking
2. **Slurm control plane** — the Slurm controller and database, running as Kubernetes pods
3. **Login nodes** — SSH-accessible entry points for submitting and managing jobs
4. **Shared storage** — a ReadWriteMany persistent volume mounted at `/home` across all nodes
5. **Topology discovery** — automatic network topology detection for optimal job placement
You then add **node sets** — groups of GPU worker nodes — to provide compute capacity. Slurm automatically discovers these nodes and makes them available for job scheduling.
The entire stack is managed by the **Crusoe Slurm Operator (CSO)**, which runs inside your cluster's control plane and keeps all Slurm components healthy and in sync.
## Key Concepts
| Concept | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Slurm Cluster** | The top-level resource. Includes the Slurm controller, login nodes, and shared storage. Created via `crusoe slurm clusters create`. |
| **Node Set** | A group of GPU worker nodes attached to a Slurm cluster. Each node set maps to an underlying CMK node pool. Created via `crusoe slurm nodesets create`. |
| **Login Node** | An SSH-accessible pod where users connect to submit and manage jobs. Multiple replicas can be configured for availability. |
| **Shared Storage** | A persistent filesystem mounted at `/home` on all login and worker nodes. Backed by Crusoe CSI. |
| **Users & Groups** | Linux users provisioned across all Slurm components via Kubernetes Custom Resources. See [User Management](./user-management.md). |
## Supported GPU Types
| GPU | Instance Type |
| -------------------- | ---------------------- |
| 8x NVIDIA B200 180GB | `b200-180gb-sxm-ib.8x` |
| 8x NVIDIA H200 141GB | `h200-141gb-sxm-ib.8x` |
| 8x NVIDIA H100 80GB | `h100-80gb-sxm-ib.8x` |
| 8x NVIDIA A100 80GB | `a100-80gb-sxm-ib.8x` |
Support for additional GPU types is coming soon.
## What's Included
When you create a Managed Slurm cluster, the following components are automatically installed and managed. You do not need to install or configure these yourself:
| Component | Purpose |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Crusoe Slurm Operator (CSO) | Manages Slurm lifecycle and configuration |
| Slinky | Runs Slurm daemons (slurmctld, slurmd) as Kubernetes pods |
| Topograph | Discovers network topology for topology-aware job scheduling |
| cert-manager | Certificate management for internal services |
| Crusoe CSI Driver | Provides shared filesystem storage |
| NVIDIA GPU Operator | GPU device plugin and drivers |
| NVIDIA Network Operator | InfiniBand and high-speed networking |
| Crusoe Load Balancer Controller | Exposes login nodes via external IP |
| Slurm Login Nodes | By default, two `c1a.8x` login nodes will be created. You can set a different login node type or a different number of login nodes. These are billable resources that are required for Slurm to run. |
| Slurm Controller Nodes | Three `c1a.4x` slurm controller nodes will be created to run the Slurm control plane. These are billable resources that are required for Slurm to run. |
## Automatic Hardware Remediation
Managed Slurm clusters include [AutoClusters](../cmk/autoclusters.md), which automatically detects critical hardware failures such as GPUs or HCAs falling off the bus. When an issue is detected, the affected node is taken down and any running jobs are cancelled and requeued to healthy nodes. The bad node is then replaced automatically. For details on how to handle this in your jobs — including the SIGTERM grace period — see [Automatic Hardware Remediation](./advanced-kubernetes.md#automatic-hardware-remediation).
## Next Steps
- [Quickstart](./quickstart.md) — Create your first Slurm cluster and run a GPU job using the Crusoe CLI
- [User Management](./user-management.md) — Add users and groups to your cluster
- [Managing Partitions](./managing-partitions.md) — Create and manage partitions in your Slurm cluster
- [Node Health Checks](./node-health-checks.md) — Built-in health checks and adding your own health/prolog/epilog checks
- [Slurm Metrics](./slurm-metrics.md) — Monitor cluster health and job performance
- [Set Up Slurm on Kubernetes](./kubernetes-setup.md) — Alternative setup path: install on a CMK cluster directly via kubectl/Helm
- [Advanced: Kubernetes Operations](./advanced-kubernetes.md) — Direct kubectl access and CRD-level configuration
- For Slurm command reference, see the [official Slurm documentation](https://slurm.schedmd.com/)
---
# Quickstart
This guide walks you through creating a Managed Slurm cluster, adding GPU worker nodes, and running your first job — all from the Crusoe CLI. You can also create Slurm clusters through the Crusoe Cloud Console by navigating to **Orchestration** > **Slurm** in the left-hand navigation pane.
## Prerequisites
- Make sure you are using the latest `crusoe` CLI version
- Reach out to customer support to check if you have high enough quotas to create Slurm clusters and associated underlying resources.
## Step 1: Create a Slurm Cluster
Create a new Managed Slurm cluster with a single command:
```sh
crusoe slurm clusters create \
--name my-slurm-cluster \
--location us-southcentral1-a \
--keyfile ~/.ssh/id_ed25519.pub \
```
This command provisions a complete Slurm environment including the underlying Kubernetes cluster, Slurm controller, login nodes, and shared storage. The required add-ons are automatically included.
:::note
This command provisions the latest supported Slurm version, currently `25.11.2-cmk.17`.
:::
**Required flags:**
| Flag | Description |
| ------------ | --------------------------------------------------------------- |
| `--name` | Name for your Slurm cluster |
| `--location` | Crusoe Cloud location (e.g., `us-east1-a`) |
| `--keyfile` | Path to your SSH public key file for root access to login nodes |
**Optional flags:**
| Flag | Default | Description |
| -------------------- | -------- | ------------------------------------------------------------------------------------------------------- |
| `--login-node-type` | `c1a.8x` | Instance type for login nodes. Only CPU types are supported. |
| `--login-replicas` | `2` | Number of login node replicas. Minimum 1, maximum 10. |
| `--home-volume-size` | `10Ti` | Shared `/home` volume size. Format: `Ti` where n \>\= 1 and n \<\= 1000 (e.g. `1Ti`, `10Ti`, `50Ti`) |
| `--subnet-id` | — | Subnet ID for the cluster |
:::note
Cluster creation typically takes around 30 minutes. The command will wait for the operation to complete and display the result.
:::
## Step 2: Check Cluster Status
Verify your cluster is running:
```sh
crusoe slurm clusters get my-slurm-cluster
```
Example output:
```
name: my-slurm-cluster
id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
state: RUNNING
location: us-southcentral1-a
login node endpoint: 160.211.64.102
login node type: c1a.8x
login replicas: 2
nodesets: []
home volume size: 10Ti
subnet id: 5efd0079-bf7e-4e0a-b879-b9af83ac3cac
root ssh pub keys: [ssh-ed25519 AAAA...]
```
Wait until the `state` field shows `RUNNING` before proceeding.
To list all your Slurm clusters:
```sh
crusoe slurm clusters list
```
## Step 3: Add GPU Worker Nodes
Add a node set to provide GPU compute capacity:
```sh
crusoe slurm nodesets create \
--name gpu-workers \
--cluster-name my-slurm-cluster \
--type h100-80gb-sxm-ib.8x \
--count 2 \
--ib-partition-id
```
**Required flags:**
| Flag | Description |
| ---------------------------------- | -------------------------------------------------------------------------------- |
| `--name` | Name for the node set |
| `--cluster-name` or `--cluster-id` | The Slurm cluster to attach to |
| `--type` | GPU instance type (see [Supported GPU Types](./overview.md#supported-gpu-types)) |
| `--count` | Number of worker nodes |
**Optional flags:**
| Flag | Description |
| ------------------- | --------------------------------------------------- |
| `--ib-partition-id` | InfiniBand partition ID for high-speed interconnect |
| `--keyfile` | Path to SSH public key file for worker node access |
| `--subnet-id` | Subnet for the node pool |
## Step 4: Verify Node Set Status
Check that your worker nodes are ready:
```sh
crusoe slurm nodesets list --cluster-name my-slurm-cluster
```
Example output:
```
name id type count state
gpu-workers b2c3d4e5-f6a7-8901-bcde-f12345678901 h100-80gb-sxm-ib.8x 2 RUNNING
```
Wait until the `state` shows `RUNNING`. You can also check a specific node set:
```sh
crusoe slurm nodesets get gpu-workers --cluster-name my-slurm-cluster
```
## Step 5: Connect to Your Cluster
Use the login node endpoint from Step 2 to SSH into your cluster:
```sh
ssh root@
```
:::tip
If you are prompted for a password, specify your private key explicitly: `ssh -i ~/.ssh/id_ed25519 root@`
:::
Once connected, verify your Slurm cluster is healthy:
```sh
sinfo
```
You should see your worker nodes in an `idle` state, ready to accept jobs.
## Step 6: Run Your First Job
### Interactive GPU Test
Run a quick interactive test to verify GPU access:
```sh
srun --gpus=8 nvidia-smi
```
This allocates a worker node with 8 GPUs and runs `nvidia-smi`, displaying GPU information.
### Batch Job
Create a batch job script named `hello-gpu.batch`:
```sh
#!/bin/bash
#SBATCH --job-name=hello-gpu
#SBATCH --nodes=1
#SBATCH --gpus-per-node=8
#SBATCH --time=5:00
#SBATCH --output=/home/hello-gpu_%j.out
srun nvidia-smi
```
:::note
Always write job output to a path under `/home`. `/home` is the shared volume mounted on every login and worker node, so output written there is visible from the login node where you run `sbatch`. If you use a relative path or any other local path, the file lands on the **worker node's** local filesystem (typically `/root` for the `root` user), and the `cat` step below will fail with `No such file or directory`.
:::
Submit the job:
```sh
sbatch hello-gpu.batch
```
Monitor the job:
```sh
squeue
```
Once the job completes, check the output:
```sh
cat /home/hello-gpu_.out
```
## Managing Node Sets
### Adding Another Node Set
You can attach multiple node sets (for example, to include different GPU types) to the same cluster:
```sh
crusoe slurm nodesets create \
--name a100-workers \
--cluster-name my-slurm-cluster \
--type a100-80gb-sxm-ib.8x \
--count 4 \
--ib-partition-id
```
### Listing Node Sets
```sh
crusoe slurm nodesets list --cluster-name my-slurm-cluster
```
### Getting Node Set Details
```sh
crusoe slurm nodesets get gpu-workers --cluster-name my-slurm-cluster
```
### Deleting a Node Set
```sh
crusoe slurm nodesets delete gpu-workers --cluster-name my-slurm-cluster
```
## Deleting a Cluster
To delete a Slurm cluster, first remove all node sets, then delete the cluster:
```sh
crusoe slurm nodesets delete gpu-workers --cluster-name my-slurm-cluster
crusoe slurm clusters delete my-slurm-cluster
```
:::warning
Deleting a cluster removes the Slurm controller, login nodes, node sets, and all Slurm state (job history, running jobs, configuration). This action cannot be undone.
:::
:::note
**The shared `/home` volume is not deleted automatically.** It is preserved so you don't lose data if you accidentally delete a cluster, and so you can recover or migrate the data afterward. To delete the volume, you must remove it manually. If you are having trouble finding the right volume to delete, contact [Crusoe Cloud Support](https://support.crusoecloud.com/) for help. Note that the volume will continue to incur storage charges until it is deleted.
:::
## Slurm Commands Reference
Once connected to a login node, use standard Slurm commands to manage jobs:
| Command | Description |
| --------- | ---------------------------------------- |
| `sinfo` | View cluster status and node information |
| `squeue` | View the job queue |
| `sbatch` | Submit a batch job |
| `srun` | Run a job interactively |
| `scancel` | Cancel a job |
For GPU jobs, specify GPU requirements using the `--gpus` flag:
```sh
srun --gpus=1 nvidia-smi # Request 1 GPU
srun --gpus=8 my-training-script # Request 8 GPUs (full node)
```
## Troubleshooting
### Common Issues
| Issue | Resolution |
| ----------------------- | ---------------------------------------------------------------------------------------------- |
| Nodes in drain state | Check node reasons with `sinfo -R` to identify configuration issues |
| GPU not detected | Check node set status via `crusoe slurm nodesets get --cluster-name ` |
| Job allocation failures | Check available resources with `sinfo` and verify job requirements are within cluster capacity |
| SSH connection refused | Ensure the cluster is in `RUNNING` state and your SSH key matches the one used during creation |
### Checking Cluster Health
From a login node:
```sh
sinfo # Check node states
scontrol show node # View detailed node information
scontrol show config # View Slurm configuration
```
## Next Steps
- [User Management](./user-management.md) — Add users and groups to your cluster
- [Managing Partitions](./managing-partitions.md) — Create and manage partitions in your Slurm cluster
- [Node Health Checks](./node-health-checks.md) — Built-in health checks and adding your own health/prolog/epilog checks
- [Slurm Metrics](./slurm-metrics.md) — Monitor cluster health and performance
- [Advanced: Kubernetes Operations](./advanced-kubernetes.md) — Direct kubectl access and CRD-level configuration
- For Slurm command reference, see the [official Slurm documentation](https://slurm.schedmd.com/)
---
# User management
Crusoe Managed Slurm allows you to create multiple Linux users with access to your Slurm cluster. Each user is provisioned across all Slurm components — login nodes, compute nodes, and the controller. Users are managed via the `SlurmUser` Kubernetes Custom Resource, and groups via the `SlurmUserGroup` Custom Resource.
:::info
CLI-based user management is coming soon. For now, users and groups are managed via `kubectl`.
:::
## Prerequisites
- A running Managed Slurm cluster (see [Quickstart](./quickstart.md))
- `kubectl` configured with access to the CMK cluster backing the Slurm cluster. Since the name of the backing CMK cluster matches the name of your Slurm cluster, the following command will give `kubectl` the correct credentials:
```sh
crusoe kubernetes clusters get-credentials
```
### Get Your Cluster Name
Run the following command to find the SlurmCluster name used as the `clusterReference` in user and group resources:
```sh
kubectl get slurmclusters -n slurm
```
Example output:
```
NAMESPACE NAME AGE
slurm my-slurm-cluster 4h25m
```
## Limitations
- By default, all users have sudo access (via the `slurm-admin` group). Set `disableSudo: true` in the user spec to remove sudo access.
- Usernames must be valid POSIX usernames: max 32 characters, start with a lowercase letter, contain only lowercase letters, digits, or hyphens, and not end with a hyphen
- UIDs are automatically assigned from the range 10000–29999. Group GIDs are assigned from the range 50000–65533.
- Changes to users or groups take approximately one minute to propagate through the cluster
## Managing Users
### Creating a User
Create a file named `user-alice.yaml`:
```yaml
apiVersion: slurm.crusoe.ai/v1alpha1
kind: SlurmUser
metadata:
name: alice # POSIX username (max 32 chars, lowercase + digits + hyphens)
namespace: slurm
spec:
clusterReference: my-slurm-cluster # Must match your SlurmCluster name (immutable after creation)
fullName: "Alice Johnson" # Optional — GECOS field
shell: /bin/bash # Optional — defaults to /bin/bash
sshPublicKeys: # One or more SSH public keys
- ssh-ed25519 AAAAC3... alice@laptop
disableSudo: false # Optional — set to true to remove sudo access (default: false)
```
Apply it:
```sh
kubectl apply -f user-alice.yaml
```
The user can now SSH into the login nodes:
```sh
ssh -i alice@
```
:::tip
Find the login node IP using `crusoe slurm clusters get ` and looking at the `login node endpoint` field. You can also find the login node IP in the UI in the Slurm cluster detail view. Alternatively:
```sh
kubectl get svc -n slurm
```
The `EXTERNAL-IP` of the login service is the login node IP.
:::
### Listing Users
```sh
kubectl get slurmusers -n slurm
```
Example output:
```
NAMESPACE NAME USERNAME UID GID CLUSTER CREATED AGE
slurm alice alice 10001 10001 my-slurm-cluster True 8s
```
### Updating a User
The `sshPublicKeys`, `fullName`, `shell`, and `disableSudo` fields can be updated after creation. The `clusterReference` field is immutable. Use the following command:
```sh
kubectl edit slurmuser -n slurm alice
```
:::tip
Use `kubectl explain slurmusers.spec` to see all available fields and their descriptions directly from the cluster.
:::
### Deleting a User
```sh
kubectl delete slurmuser -n slurm alice
```
The user immediately loses access to the Slurm cluster.
## Managing Groups
`SlurmUserGroup` resources let you organize users into groups for features like partition access control.
### Creating a Group
Create a file named `group-ml-team.yaml`:
```yaml
apiVersion: slurm.crusoe.ai/v1alpha1
kind: SlurmUserGroup
metadata:
name: ml-team # POSIX group name (max 32 chars, same naming rules as users)
namespace: slurm
spec:
clusterReference: my-slurm-cluster # Must match your SlurmCluster name (immutable after creation)
members: # List of SlurmUser names
- alice
- bob
```
Apply it:
```sh
kubectl apply -f group-ml-team.yaml
```
:::note
Users must reconnect their SSH session to pick up new group membership.
:::
### Listing Groups
```sh
kubectl get slurmusergroups -n slurm
```
Example output:
```
NAMESPACE NAME GROUPNAME GID MEMBERS CLUSTER CREATED AGE
slurm ml-team ml-team 50000 2 my-slurm-cluster True 80s
```
### Updating a Group
Add or remove users by editing the `spec.members` list:
```sh
kubectl edit slurmusergroup -n slurm ml-team
```
### Deleting a Group
```sh
kubectl delete slurmusergroup -n slurm ml-team
```
## Next Steps
- [Quickstart](./quickstart.md) — Set up your Slurm cluster
- [Managing Partitions](./managing-partitions.md) — Create and manage partitions in your Slurm cluster
- [Node Health Checks](./node-health-checks.md) — Built-in health checks and adding your own health/prolog/epilog checks
- [Slurm Metrics](./slurm-metrics.md) — Monitor cluster health and job performance
- [Advanced: Kubernetes Operations](./advanced-kubernetes.md) — Direct kubectl access and CRD-level configuration
- For Slurm command reference, see the [official Slurm documentation](https://slurm.schedmd.com/)
---
# Managing Partitions
Partitions control which nodes are available to specific groups of users and set resource limits. Partitions are defined in the Slinky Controller CR's `spec.extraConf` field.
Target nodes by **node set**, not by node name. Every node set in your cluster is automatically published to Slurm as a nodeset you can reference directly in a partition's `Nodes=` field, and that reference keeps working as nodes are replaced, added, or removed.
## Why Not Node Names
Node names are not stable. When a node is replaced, its replacement comes back under a new name, and a partition that still lists the old name can stop the Slurm controller from starting. That affects scheduling across the whole cluster, not just the one partition.
:::warning
Do not list node names or ranges such as `Nodes=np-2e2792bc-[1-2]` in a partition. Use a node set name or `Nodes=ALL`, both of which stay valid as nodes come and go.
:::
## Find Your Node Set Names
Each node set is published to Slurm under the name `-`. A node set called `h200-workers` on a cluster called `research` is available to partitions as `research-h200-workers`.
To list the names available on your cluster, run this from a login node:
```sh
sinfo -h -o "%f" | sort -u
```
```
research-cpu-workers
research-h200-workers
```
Each line is a name you can use in `Nodes=`. To see which nodes belong to each one, add the node list:
```sh
sinfo -h -o "%f %N" | sort -u
```
```
research-cpu-workers np-7e02223a-1
research-h200-workers np-2e2792bc-[1-2]
```
## Creating a Partition
**Step 1 — Edit the Controller CR**
The Controller CR is named `slurm-` in the `slurm` namespace:
```sh
kubectl -n slurm edit controller slurm-
```
**Step 2 — Add partition lines to `spec.extraConf`**
The `spec.extraConf` field contains both your custom configuration and an automatically injected section managed by the Crusoe Slurm Operator. Add your `PartitionName=` lines **above** the injected section markers:
```yaml
spec:
extraConf: |
PartitionName=ml-team Nodes=research-h200-workers MaxTime=08:00:00 State=UP
# THE FOLLOWING SETTINGS ARE AUTOMATICALLY INJECTED BY CRUSOE SLURM OPERATOR
# ===============================START======================================
SlurmctldDebug=debug5
SlurmdDebug=debug5
...
PartitionName=all Nodes=ALL Default=YES MaxTime=UNLIMITED State=UP
# ================================END=======================================
```
:::warning
Do not modify anything between the `START` and `END` markers. The operator overwrites this section on every reconciliation cycle.
:::
:::note
If you add `Default=YES` to your custom partition, the operator will automatically remove `Default=YES` from the `all` partition in the injected section.
:::
**Step 3 — Verify the partition**
From a login node, run `sinfo` to confirm the new partition is available:
```
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
all* up infinite 3 idle np-2e2792bc-[1-2],np-7e02223a-1
ml-team up 8:00:00 2 idle np-2e2792bc-[1-2]
```
**Step 4 — Reconfigure Slurm (optional)**
If Slurm doesn't pick up the change automatically, run from a login node:
```sh
scontrol reconfigure
```
## Splitting Compute Across Partitions
A node set is the smallest unit of partition membership. You cannot split one node set between two partitions and have the split survive node replacement, so plan node sets around how you intend to partition compute. If a team or workload needs its own partition, give it its own node set.
Point each partition at the node set that backs it:
```
PartitionName=ml-team Nodes=research-h200-workers MaxTime=08:00:00 State=UP
PartitionName=cpu-tasks Nodes=research-cpu-workers MaxTime=UNLIMITED State=UP
```
Several partitions can share one node set, which is the usual way to offer the same hardware at different priorities:
```
PartitionName=high Nodes=research-h200-workers Default=NO MaxTime=UNLIMITED State=UP PriorityTier=100
PartitionName=normal Nodes=research-h200-workers Default=YES MaxTime=UNLIMITED State=UP PriorityTier=10
PartitionName=low Nodes=research-h200-workers Default=NO MaxTime=UNLIMITED State=UP PriorityTier=1
```
A partition can also span several node sets. List them comma separated, and the partition contains every node from each:
```
PartitionName=everything Nodes=research-h200-workers,research-cpu-workers MaxTime=UNLIMITED State=UP
```
:::note
A partition that references a node set with no registered nodes is not an error. It appears in `sinfo` with zero nodes and starts scheduling as soon as nodes join. This is what makes node set references safe to configure before capacity is attached.
:::
## Next Steps
- [Quickstart](./quickstart.md) — Set up your Slurm cluster
- [User Management](./user-management.md) — Create and manage users and groups
- [Slurm Metrics](./slurm-metrics.md) — Monitor cluster health and job performance
- [Node Health Checks](./node-health-checks.md) — Built-in health checks and adding your own health/prolog/epilog checks
- [Advanced: Kubernetes Operations](./advanced-kubernetes.md) — Direct kubectl access and CRD-level configuration
- For Slurm command reference, see the [official Slurm documentation](https://slurm.schedmd.com/)
---
# Node Health Checks
Managed Slurm ships a built-in node health-check suite that runs automatically at three points in the job lifecycle:
- **Periodic** — every 5 minutes on idle nodes (`HealthCheckProgram`)
- **Pre-job (prolog)** — on each allocated node before a job starts
- **Post-job (epilog)** — on each allocated node after a job ends
Crusoe wires these dispatchers into `slurm.conf` for you. To add your own checks, use the `SlurmClusterHealthCheck` (SCHC) custom resource described below. **Do not set your own `Prolog`, `Epilog`, or `HealthCheckProgram` in the Controller's `spec.extraConf`** — they would conflict with the managed dispatchers.
When a check fails it either **drains** the node (no new jobs are scheduled; the reason is visible in `sinfo -R` and `scontrol show node`) or logs a **warning** (no scheduling impact). A drain during the prolog also requeues the in-flight job. Check results are surfaced in each worker pod's `logfile` container (view with `kubectl logs`); prolog and epilog output is additionally written to `/var/log/prolog/.log` and `/var/log/epilog/.log` on the node where they run.
## Prerequisites
- A running Managed Slurm cluster (see [Quickstart](./quickstart.md))
- `kubectl` configured with access to the CMK cluster backing the Slurm cluster. Since the name of the backing CMK cluster matches the name of your Slurm cluster, the following command will give `kubectl` the correct credentials:
```sh
crusoe kubernetes clusters get-credentials
```
### Get Your Cluster Name
Run the following command to find the SlurmCluster name used in the health-check object names:
```sh
kubectl get slurmclusters -n slurm
```
Example output:
```
NAMESPACE NAME AGE
slurm my-slurm-cluster 4h25m
```
## Limitations
- Add your own checks to the `-custom` object only. The `-defaults` object is Crusoe-managed and may be overwritten.
- Each object holds at most 50 scripts, and each script's `source` is limited to 16 KB.
- Script names must match `NN-name` (a two-digit prefix followed by lowercase letters, digits, and hyphens) and be unique within the object. The prefix only orders your scripts relative to each other; they always run after the built-in checks.
- To drain a node from a check, call `scontrol` directly (see [Adding a Check](#adding-a-check)).
- Changes take a few seconds to propagate to all nodes and apply on the next run of the affected phase.
## What the Built-in Checks Cover
| Phase | When it runs | What it checks (examples) |
| ------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Periodic (`checks`) | Every 5 min on idle nodes | `/home` and `/data` NFS mount, writability, and free space; GPU count vs. registered Slurm GRES; DCGM GPU health; NVIDIA driver persistence mode; InfiniBand port link state; system load, available memory/swap, local `/tmp` and `/dev/shm` space; kernel errors in `dmesg` (MCE, EDAC, disk I/O, NFS timeouts) |
| Pre-job (`prolog`) | Before a job, on each allocated node | required `SLURM_*` job variables present; residual GPU memory below threshold; `CUDA_VISIBLE_DEVICES` count matches allocated GPUs; DCGM level-1 diagnostics |
| Post-job (`epilog`) | After a job, on each allocated node | ECC double-bit errors recorded during the job; prune stale Docker containers; reset GPUs left with residual memory; kill leftover job processes and clean up orphaned shared memory; then re-run the periodic checks |
For the full list of built-in checks — including the exact failure trigger, whether it warns or drains the node, and when each is skipped — see the [Default Check Reference](#default-check-reference) at the bottom of this page.
## Managing Your Checks
The health-check suite is managed by the `SlurmClusterHealthCheck` CRD (short name `schc`). Each cluster has two objects in the `slurm` namespace:
- `-defaults` — the Crusoe-provided checks. Managed by CSO; do not edit.
- `-custom` — empty by default and **never overwritten by CSO**. Add your own checks here.
Your scripts run automatically **after** the built-in scripts for the matching phase. Each script runs independently, so a crash or unhandled error in one is logged as a warning and does not affect the other checks.
:::tip
Use `kubectl explain slurmclusterhealthchecks.spec` to see all available fields and their descriptions directly from the cluster.
:::
### Adding a Check
Add a script by editing the `-custom` object:
```sh
kubectl edit schc -custom -n slurm
```
```yaml
apiVersion: slurm.crusoe.ai/v1alpha1
kind: SlurmClusterHealthCheck
metadata:
name: -custom
namespace: slurm
spec:
scripts:
# Periodic check — runs every 5 minutes on idle nodes
- name: "60-scratch-mount"
type: checks
enabled: true
source: |
#!/usr/bin/env bash
if ! mountpoint -q /mnt/scratch; then
echo "$(date -u +%FT%TZ) scratch_mount FAIL host=$(hostname -s) reason=not-mounted" >&2
scontrol update NodeName="$(hostname -s)" State=DRAIN Reason="scratch: /mnt/scratch not mounted"
fi
exit 0
# Pre-job check — runs before every job on each allocated node
- name: "60-dataset-present"
type: prolog
enabled: true
source: |
#!/usr/bin/env bash
if [[ ! -r /data/shared/dataset.bin ]]; then
echo "$(date -u +%FT%TZ) dataset_present FAIL host=$(hostname -s) job=${SLURM_JOB_ID} reason=dataset-unreadable" >&2
scontrol update NodeName="$(hostname -s)" State=DRAIN Reason="prolog: shared dataset unreadable"
exit 1 # drains the node and requeues the job on a healthy node
fi
exit 0
# Post-job check — runs after every job on each allocated node
- name: "60-scratch-cleanup"
type: epilog
enabled: true
source: |
#!/usr/bin/env bash
rm -rf "/mnt/scratch/job-${SLURM_JOB_ID}" || \
echo "$(date -u +%FT%TZ) scratch_cleanup WARN host=$(hostname -s) reason=cleanup-failed" >&2
exit 0 # epilog must always exit 0
```
**Field reference:**
- `name` — script filename in the form `NN-name` (a two-digit prefix, then lowercase letters, digits, and hyphens). The prefix orders your scripts relative to each other within a phase; any number works, since your scripts always run after the built-in checks.
- `type` — one of `checks`, `prolog`, or `epilog`.
- `enabled` — set to `false` to keep a script defined but exclude it from execution.
- `source` — the shell script. Write to **stderr** for log output (surfaced in the worker pod's `logfile` container via `kubectl logs`).
**Available environment variables:**
- Periodic (`checks`) scripts run with no job context — no `SLURM_JOB_*` variables are set.
- `prolog` and `epilog` scripts run with the standard Slurm job environment, including `SLURM_JOB_ID`, `SLURM_JOB_USER`, `SLURM_JOB_UID`, `SLURM_JOB_NODELIST`, and — for GPU jobs — `CUDA_VISIBLE_DEVICES` and `SLURM_JOB_GPUS`.
**Draining and exit codes:**
- To drain the node, call `scontrol` directly: `scontrol update NodeName="$(hostname -s)" State=DRAIN Reason=""` (keep the reason under 250 characters).
- `checks` and `prolog` scripts should `exit 0` after draining — the drain is signalled by the `scontrol` call, not the exit code. In a `prolog` script, `exit 1` instead if you also want the job requeued.
- `epilog` scripts **must always `exit 0`** — Slurm treats a non-zero epilog exit as a node-fatal event.
- For a warn-only check, just log to stderr and `exit 0`.
Changes are picked up automatically across all nodes within seconds of saving (no `scontrol reconfigure` needed) and take effect on the next run of that phase.
### Viewing Your Checks
List the health-check objects for your cluster:
```sh
kubectl get schc -n slurm
```
To see the scripts currently defined in your custom object:
```sh
kubectl get schc -custom -n slurm -o yaml
```
The scripts are mounted on each worker pod under `/opt/crusoe/healthcheck`, so you can inspect what's running on a node directly:
```sh
kubectl exec -n slurm -c slurmd -- ls /opt/crusoe/healthcheck/
```
### Removing a Check
Edit the `-custom` object and delete the script's entry from `spec.scripts` (or set `enabled: false` to keep it defined but inactive):
```sh
kubectl edit schc -custom -n slurm
```
The script is removed from the affected nodes within seconds.
## Per-Job Task Prolog/Epilog (User-Configured)
Users can specify prolog and epilog scripts per job using `--task-prolog` and `--task-epilog`. No admin setup is required.
In an sbatch script:
```sh
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --task-prolog=/home/alice/setup.sh
#SBATCH --task-epilog=/home/alice/teardown.sh
srun ./myprogram
```
Or inline with srun:
```sh
srun --task-prolog=/home/alice/setup.sh --task-epilog=/home/alice/teardown.sh ./myprogram
```
**Key behaviors:**
- Runs **once per task** (not per node), as the submitting user
- stdout appears in the job's output file
- Non-zero exit logs a warning but does **not** requeue the job
- No admin involvement required
## Execution Order
When both the managed health-check prolog/epilog and per-job task scripts are configured, each allocated node runs them in this order:
| Step | What runs | Runs as | Scope |
| ---- | ----------------------------------------------------------------------------- | --------------- | -------- |
| 1 | **Prolog dispatcher** — built-in ambient checks, then your `checks` scripts | root | Per node |
| 2 | **Prolog dispatcher** — built-in pre-job checks, then your `prolog` scripts | root | Per node |
| 3 | `--task-prolog` (if set on the job) | Submitting user | Per task |
| 4 | The job itself (`srun ./myprogram`) | Submitting user | Per task |
| 5 | `--task-epilog` (if set on the job) | Submitting user | Per task |
| 6 | **Epilog dispatcher** — built-in cleanup, then your `epilog` scripts | root | Per node |
| 7 | **Epilog dispatcher** — built-in ambient re-check, then your `checks` scripts | root | Per node |
## Default Check Reference
The built-in checks shipped in the `-defaults` object, grouped by phase. GPU and InfiniBand checks skip cleanly on nodes without that hardware. Thresholds shown are the defaults. Changes to default checks (disabling checks, adjusting thresholds, changing failure behavior) can be done by reaching out to our support team.
### Periodic Checks
Run every 5 minutes on idle nodes.
| Check | What it verifies | On failure | Skipped when |
| -------------------------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| `10-filesystem` | `/home` is NFS-mounted and writable | **Warn** — not NFS-mounted, or the write probe fails | — |
| `11-filesystem-data` | `/data` is NFS-mounted and writable | **Warn** — not NFS-mounted, or the write probe fails | `/data` is absent on the node |
| `12-filesystem-space` | `/home` has at least 50 GB free | **Warn** — free space below threshold, or `df` fails | — |
| `13-filesystem-data-space` | `/data` has at least 50 GB free | **Warn** — free space below threshold, or `df` fails | `/data` is absent on the node |
| `15-hardware` | GPU count (from Slurm GRES and `nvidia-smi`) matches the expected count (8) | **Drain** — GPU count below expected; **Warn** — GRES unavailable or the `nvidia-smi` query fails | GPU portion skipped on non-GPU nodes |
| `20-gpu-dcgm` | DCGM reports all GPUs healthy | **Drain** — DCGM overall health is a failure; **Warn** — DCGM reports a warning-level event, or DCGM is unreachable | Non-GPU node |
| `25-network-ib` | All InfiniBand ports are in the `Active` link state | **Drain** — any port is not `Active` | The node has no InfiniBand ports |
| `30-driver` | NVIDIA persistence mode is enabled | **Drain** — persistence mode is not enabled | Non-GPU node |
| `35-load` | 1-minute load average is within 2× the CPU count | **Warn** — load above threshold | — |
| `40-fs-local` | `/tmp` and `/dev/shm` have at least 1 GB free and 10,000 free inodes | **Drain** — free space below threshold; **Warn** — free inodes below threshold | — |
| `45-memory` | At least 2 GB of memory is available and swap is not in use | **Warn** — low available memory, or swap in active use | — |
| `50-dmesg` | Kernel ring buffer is free of fatal hardware errors | **Drain** — machine-check exception, uncorrectable memory (EDAC) error, disk I/O error, or a recent NFS "server not responding"; **Warn** — correctable memory (EDAC) error | The ring buffer is empty |
### Pre-job Checks (Prolog)
Run on each allocated node before the job starts. A failure drains the node **and requeues the job**.
| Check | What it verifies | On failure | Skipped when |
| ----------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `10-job-env` | Required job variables (`SLURM_JOB_ID`, `SLURM_JOB_USER`, `SLURM_JOB_UID`) are set | **Drain + requeue** — any variable is missing | — |
| `20-gpu-residual` | Assigned GPUs hold less than 100 MiB of residual memory | **Drain + requeue** — any assigned GPU is over threshold | Non-GPU node, or no GPUs assigned to the job |
| `25-cuda-visible` | `CUDA_VISIBLE_DEVICES` count matches the GPUs allocated to the job | **Drain + requeue** — visible count is below the allocated count | Non-GPU node, no GPUs assigned, or the allocated count cannot be determined |
| `30-dcgm-diag` | DCGM level-1 diagnostic passes on the assigned GPUs | **Drain + requeue** — the diagnostic fails or times out | Non-GPU node, no GPUs assigned, or a diagnostic is already running on the node |
### Post-job Checks (Epilog)
Run on each allocated node after the job ends. Epilog never fails the job via its exit code — health failures drain the node via `scontrol` instead.
| Check | What it does | On failure | Skipped when |
| ------------------ | ------------------------------------------------------------------------ | -------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `10-dcgm-stats` | Checks job-scoped DCGM stats for ECC double-bit errors | **Drain** — one or more ECC double-bit errors were recorded during the job | Non-GPU node, no job ID, or stats are unavailable |
| `20-containers` | Prunes stopped Docker containers idle for more than 1 hour | **Warn** — the prune command fails | — |
| `30-gpu-reset` | Resets GPUs that still hold residual memory after the job | **Warn** — the reset fails | Non-GPU node, or no GPUs assigned |
| `40-processes-ipc` | Kills leftover job processes and removes orphaned shared-memory segments | **Warn** — lingering processes were found and killed (informational) | No job context, or the job user is a system account (UID < 1000) |
## Next Steps
- [Quickstart](./quickstart.md) — Create your first Slurm cluster
- [User Management](./user-management.md) — Add users and groups, manage partitions
- [Managing Partitions](./managing-partitions.md) — Create and manage partitions in your Slurm cluster
- [Slurm Metrics](./slurm-metrics.md) — Monitor cluster health and performance
- [Advanced: Kubernetes Operations](./advanced-kubernetes.md) — Direct kubectl access and CRD-level configuration
- For Slurm command reference, see the [official Slurm documentation](https://slurm.schedmd.com/)
---
# Managing Volumes
Managed Slurm clusters include a built-in `/home` shared volume available to all users. You can attach up to 2 additional Crusoe filesystem disks via the `spec.volumes` field of your `SlurmCluster` resource. Each disk is mounted on every login and worker pod at a user-defined path. Volumes can be declared when the cluster is first created or added to an existing cluster at any time.
:::warning
The `existingDiskID` field only accepts **Crusoe shared-volume** (filesystem) disks. Persistent disks are not supported. Verify a disk's type in the [Crusoe console](https://console.crusoecloud.com) or via the Crusoe CLI before using it.
:::
## Prerequisites
- A Managed Slurm cluster, or an initial `SlurmCluster` manifest if creating a new one
- `kubectl` configured with cluster credentials (see [Kubernetes Setup](./kubernetes-setup.md))
- For existing-disk mode: the UUID of a pre-existing Crusoe shared-volume disk
## Volume Modes
Each entry in `spec.volumes` must specify **exactly one** of the following:
**Dynamic provisioning (`size`)** — CSO provisions a new Crusoe filesystem disk on your behalf. The disk is created when the volume entry is added and its lifecycle is managed by CSO.
**Existing disk (`existingDiskID`)** — You provide the UUID of a pre-existing Crusoe shared-volume disk. CSO mounts the disk but never deletes it, regardless of any `reclaimPolicy` setting.
## Adding Volumes at Cluster Creation
Include a `volumes` block in your initial `SlurmCluster` manifest:
```yaml
apiVersion: slurm.crusoe.ai/v1alpha1
kind: SlurmCluster
metadata:
name: my-cluster
namespace: slurm
spec:
# ... other fields ...
volumes:
# Dynamic: CSO provisions a new 2 TiB disk
- name: datasets
size: 2Ti
mountPath: /mnt/datasets
reclaimPolicy: Retain # optional; Retain is the default
# Existing: mount a pre-existing shared-volume disk by UUID
- name: shared-data
existingDiskID: 00000000-0000-0000-0000-000000000001
mountPath: /mnt/shared
```
## Adding Volumes to an Existing Cluster
Edit your cluster's spec directly with:
```sh
kubectl edit slurmcluster -n slurm
```
Add entries to `spec.volumes` using the same format shown above. Save the file to apply the change.
:::note
Adding a volume entry triggers a rolling restart of all login and worker pods. Pods are restarted one at a time; each new pod will have the disk mounted at the specified path. Running jobs may be interrupted during the restart.
:::
## Constraints and Immutability Rules
| Field | Mutable? | Rules |
| ---------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| `name` | Immutable after creation | Used to name Kubernetes PVC/PV resources. Must be unique within the cluster. Lowercase alphanumeric and hyphens only. |
| `mountPath` | Immutable after creation | Cannot be `/home` (reserved). Cannot be a sub-path or super-path of another volume's `mountPath`. |
| `existingDiskID` | Immutable after creation | Must be a valid Crusoe shared-volume UUID. |
| `size` | Can only be increased | Must be a whole number between `1Ti` and `10Ti`. Decreasing the value is rejected. |
| `reclaimPolicy` | Mutable | Applies to dynamic volumes only. `Retain` (default) keeps the disk after the volume is removed; `Delete` permanently deletes it. |
Additional limits:
- Maximum **2 volumes** per cluster.
- `mountPath` values cannot overlap — no entry's path may be a sub-path or super-path of another entry's path.
## Checking Volume Status
After adding a volume, check its provisioning state:
```sh
kubectl get slurmcluster -n slurm -o yaml
```
Look for `status.volumesStatus` in the output:
```yaml
status:
volumesStatus:
- name: datasets
diskID: 00000000-0000-0000-0000-000000000002
diskName: datasets
diskSource: crusoe-slurm
mountPath: /mnt/datasets
phase: Ready
- name: shared-data
diskID: 00000000-0000-0000-0000-000000000001
diskName: shared-data
diskSource: customer
mountPath: /mnt/shared
phase: Ready
```
Each entry reports:
| Field | Description |
| ------------ | -------------------------------------------------------------------------------- |
| `name` | Matches the `name` from `spec.volumes` |
| `diskID` | Crusoe disk UUID (CSO-assigned for `crusoe-slurm`, user-provided for `customer`) |
| `diskName` | Name of the Crusoe disk |
| `diskSource` | `crusoe-slurm` (CSO-provisioned) or `customer` (user-provided) |
| `mountPath` | Path where the disk is mounted in login and worker pods |
| `phase` | Lifecycle state: `Provisioning`, `Ready`, `Error`, `Deleting`, `Deleted` |
| `message` | Error details when `phase` is `Error` |
A newly added volume starts in `Provisioning` and transitions to `Ready` when the PV and PVC are successfully created. If provisioning fails, the phase shows `Error` with details in `message`. When a volume entry is removed, the phase passes through `Deleting` → `Deleted`.
## Removing a Volume
Remove the corresponding entry from `spec.volumes`:
```sh
kubectl edit slurmcluster -n slurm
```
Delete the entry and save. This triggers a rolling restart of login and worker pods; the disk will no longer be mounted after the restart completes.
**What happens to the underlying disk:**
- **Dynamic volume, `reclaimPolicy: Delete`** — the Crusoe disk is permanently deleted.
- **Dynamic volume, `reclaimPolicy: Retain`** (default) — the disk is kept in your account and continues to accrue storage charges.
- **Existing disk** — the disk is never deleted by CSO, regardless of any settings.
:::note
If you resize a Crusoe shared-volume disk outside of Slurm (for example, via the Crusoe console or CLI), the change is not automatically reflected in the cluster's PVC. The pods retain access to the full disk capacity, but the PVC may display the previous size.
:::
## Next Steps
- [Quickstart](./quickstart.md) — Set up your Slurm cluster
- [Managing Partitions](./managing-partitions.md) — Configure partitions for node groups
- [Slurm Metrics](./slurm-metrics.md) — Monitor cluster health and job performance
- [Advanced: Kubernetes Operations](./advanced-kubernetes.md) — Direct kubectl access and CRD-level configuration
---
# Set up Slurm on Kubernetes
This guide walks through setting up Managed Slurm as a pure Kubernetes add-on on an existing [Crusoe Managed Kubernetes (CMK)](../cmk/overview.md) cluster. Use this approach if you want full control over the underlying CMK cluster, want to manage Slurm components alongside other Kubernetes workloads, or already have GitOps/Helm-based infrastructure tooling that you want to extend to Slurm.
If you'd rather have Crusoe provision and manage everything end-to-end through a single command, see the [Quickstart](./quickstart.md) instead — it uses the `crusoe slurm` CLI to create the cluster, login nodes, and worker pools as a managed bundle. The two paths converge to the same end state; this guide just gives you direct kubectl/Helm access at every step.
:::note
In this mode you create the CMK cluster and your GPU compute node pools yourself, and all Slurm configuration lives in Kubernetes Custom Resources. You do **not** use the `crusoe slurm clusters create` or `crusoe slurm nodesets create` commands. CSO provisions the controller and login node pools automatically as part of `SlurmCluster` reconciliation.
:::
## How It Works
The Crusoe Slurm Operator (CSO) runs inside your CMK cluster and reconciles a `SlurmCluster` Custom Resource. When you apply the CR, CSO automatically provisions and configures:
- The controller node pool (3 × `c1a.4x` for HA — managed by CSO, not user-configurable; **cannot be scaled down**)
- The login node pool (size and instance type configurable in the CR)
- The Slinky operator (Slurm-on-Kubernetes from SchedMD)
- The Slurm controller (`slurmctld`) and login pods
- Topograph for topology-aware scheduling
- The shared `/home` PersistentVolumeClaim
- A LoadBalancer service for SSH access to login nodes
For **compute workers**, you create the GPU node pools yourself — CSO can't infer hardware type or scale automatically. The operator then watches for Kubernetes nodes labeled `slurm.crusoe.ai/compute-node-type=true` and **automatically creates Slinky NodeSets** for each underlying node pool.
Users and groups are managed via `SlurmUser` and `SlurmUserGroup` CRs, the same as in the CLI/API path. See [User Management](./user-management.md) for the full reference.
## Prerequisites
- The Crusoe CLI (latest version) installed and authenticated, used here only for CMK cluster and nodepool creation
- `kubectl` installed
- `helm` v3.11+ installed
- An SSH public key for accessing login nodes
## Step 1: Create the CMK Cluster
Create a Managed Kubernetes cluster with the Slurm-supporting add-ons enabled. The required add-ons are:
- `crusoe_managed_slurm` — installs the Crusoe Slurm Operator (CSO) on the cluster
- `nvidia_gpu_operator` — exposes GPUs as schedulable resources
- `nvidia_network_operator` — enables InfiniBand networking
- `crusoe_csi` — provides the shared `/home` filesystem
- `autoclusters` — automatic hardware remediation (recommended for production)
```sh
crusoe kubernetes clusters create \
--name my-slurm-cluster \
--location us-southcentral1-a \
--add-ons crusoe_managed_slurm,nvidia_gpu_operator,nvidia_network_operator,crusoe_csi,autoclusters
```
Including `crusoe_managed_slurm` in `--add-ons` installs CSO automatically — you do not need to install it separately via Helm. CSO in turn installs cert-manager, the Slinky operator, Topograph, and the Crusoe Load Balancer Controller as it reconciles your `SlurmCluster` CR in Step 3.
Once the cluster is `RUNNING`, configure `kubectl`:
```sh
crusoe kubernetes clusters get-credentials my-slurm-cluster
kubectl cluster-info
```
## Step 2: Create the `slurm` Namespace
All Slurm Custom Resources live in a single namespace. Only one `SlurmCluster` CR is supported per namespace.
```sh
kubectl create namespace slurm
```
## Step 3: Apply the `SlurmCluster` Custom Resource
Applying a `SlurmCluster` CR triggers CSO to install the full Slurm stack — cert-manager, the Slinky operator, Topograph, the Crusoe Load Balancer Controller, the Slurm controller, login pods, and the shared `/home` volume. The controller node pool (3 `c1a.4x nodes`), login node pool (configurable in CRD below), and shared `/home` volume (configurable in CRD below) are provisioned automatically as part of this step.
Create a file named `slurm-cluster.yaml`:
```yaml
apiVersion: slurm.crusoe.ai/v1alpha1
kind: SlurmCluster
metadata:
name: my-slurm-cluster
namespace: slurm
spec:
clusterVersion: "25.11.2-cmk.17"
loginSet:
replicas: 2
instanceType: c1a.8x
# Optional. CIDR ranges allowed to reach the login LoadBalancer.
# Defaults to 0.0.0.0/0 if omitted.
firewallRuleSourceRanges:
- 0.0.0.0/0
rootSSHPubKeys:
- "ssh-ed25519 AAAA... user@host"
# Shared /home volume. Default 10Ti. Can be increased later but never decreased.
homeVolumeSize: "10Ti"
```
Apply it:
```sh
kubectl apply -f slurm-cluster.yaml
```
### Spec Reference
| Field | Required | Default | Notes |
| ----------------------------------- | -------- | ----------- | ---------------------------------------------------------------------------------------------------------------------- |
| `clusterVersion` | Yes | — | Slurm version to install. Currently, the only supported version is `25.11.2-cmk.17`. |
| `loginSet.replicas` | Yes | — | Number of login pods |
| `loginSet.instanceType` | No | `c1a.8x` | Login node instance type. Immutable after creation. |
| `loginSet.firewallRuleSourceRanges` | No | `0.0.0.0/0` | CIDR ranges allowed to reach the login LoadBalancer service |
| `rootSSHPubKeys` | Yes | — | SSH public keys authorized for the `root` user on login and worker nodes |
| `homeVolumeSize` | No | `10Ti` | Shared `/home` volume size. Format `Ti` (e.g. `10Ti`, `50Ti`). Can be increased after creation but never decreased. |
### Watching the Reconciliation
Cluster bring-up takes ~10 minutes once the node pools are ready. Watch the phases:
```sh
kubectl get slurmclusters -n slurm -w
```
The `STATUS` column moves through `Provisioning` → `Installing` → `Ready`. For detail on which component is currently being configured:
```sh
kubectl describe slurmcluster my-slurm-cluster -n slurm
```
The conditions list (`CertManagerReady`, `SlinkyReady`, `TopographReady`, `ControllerReady`, `LoginReady`, etc.) indicates progress.
Once the cluster shows `Ready`, the `LOGIN` column on `kubectl get slurmclusters` displays `2/2` and `ENDPOINT` shows the external IP of the login LoadBalancer.
At this point you have a healthy Slurm control plane with login nodes, but no compute capacity yet. Add compute node pools next.
## Step 4: Add Compute Node Pools
Compute pools provide the GPU capacity that runs your jobs. Unlike controller and login pools, compute pools are **not** auto-created by CSO — you create them yourself so you can choose the GPU type, count, and InfiniBand configuration. You can create multiple pools (for example, one per GPU type) and the operator will create a Slinky NodeSet for each one automatically. CPU node pools can also be created, but do not need the `--ib-partition-id` flag set.
Each compute pool needs the `slurm.crusoe.ai/compute-node-type=true` label and an InfiniBand partition ID:
```sh
crusoe kubernetes nodepools create \
--name slurm-h100-workers \
--cluster-name my-slurm-cluster \
--type h100-80gb-sxm-ib.8x \
--count 4 \
--ib-partition-id \
--node-labels "slurm.crusoe.ai/compute-node-type=true"
```
CMK automatically applies `crusoe.ai/nodepool.id` and `crusoe.ai/nodepool.name` labels to every node based on the parent node pool. The Slurm operator uses the `nodepool.id` label to group compute nodes into NodeSets — one NodeSet per pool.
For clusters with the Slurm addon, all NVMe storage is striped by default for use by containerd.
:::tip
You can add or remove compute pools at any time. The operator detects new pools via the node labels and creates additional NodeSets without any CR changes on your part.
:::
### Verify Compute Nodes Were Discovered
The operator watches for compute nodes and creates a Slinky NodeSet for each unique `crusoe.ai/nodepool.id`:
```sh
kubectl get nodesets -n slurm
```
You should see one entry per compute node pool, with `READY` showing `/` once Slurm has registered all the nodes.
You can also verify the labels on your compute nodes directly:
```sh
kubectl get nodes -L slurm.crusoe.ai/compute-node-type,crusoe.ai/nodepool.id
```
Each compute node should show `true` for `compute-node-type` and a non-empty `nodepool.id`.
## Step 5: Add Users (Optional)
To allow non-root users to SSH in and submit jobs, apply `SlurmUser` and `SlurmUserGroup` CRs. The full reference, including UID/GID assignment and POSIX naming rules, is in [User Management](./user-management.md). Brief example:
```yaml
apiVersion: slurm.crusoe.ai/v1alpha1
kind: SlurmUser
metadata:
name: alice
namespace: slurm
spec:
clusterReference: my-slurm-cluster
fullName: "Alice Johnson"
sshPublicKeys:
- "ssh-ed25519 AAAA... alice@laptop"
```
```sh
kubectl apply -f user-alice.yaml
```
The user can SSH in within ~1 minute:
```sh
ssh -i alice@
```
You can find the login node endpoint with:
```sh
kubectl get slurmcluster my-slurm-cluster -n slurm -o jsonpath='{.status.loginEndpoint}'
```
## Step 6: Run a Job
Once SSH'd into a login node, standard Slurm commands work as usual:
```sh
sinfo # Show available nodes
srun --gpus=8 nvidia-smi # Quick interactive GPU test
sbatch my-job.batch # Submit a batch job
squeue # Check the queue
```
For a multi-node NCCL test and other examples, see [Advanced: Kubernetes Operations — Running NCCL Tests](./advanced-kubernetes.md#running-nccl-tests).
## Adding More Compute Pools
To add capacity, simply create another node pool with the compute label:
```sh
crusoe kubernetes nodepools create \
--name slurm-a100-workers \
--cluster-name my-slurm-cluster \
--type a100-80gb-sxm-ib.8x \
--count 4 \
--ib-partition-id \
--node-labels "slurm.crusoe.ai/compute-node-type=true"
```
Within a few seconds, a new Slinky NodeSet appears in the `slurm` namespace and the new nodes register with `slurmctld`. Run `sinfo` from a login node to confirm.
## Updating the Cluster
Most spec fields on `SlurmCluster` are immutable after creation. The supported updates are:
- `homeVolumeSize` — increase only (PVC limitation)
- `loginSet.replicas` — scale up or down within the available login node count
- `rootSSHPubKeys` — propagates to running login and worker pods within ~1 minute
- `loginSet.firewallRuleSourceRanges` — updates the LoadBalancer firewall
Apply changes with `kubectl apply -f slurm-cluster.yaml` or `kubectl edit slurmcluster my-slurm-cluster -n slurm`.
## Deleting the Cluster
Delete the `SlurmCluster` CR. The operator will tear down all components and clean up the LoadBalancer:
```sh
kubectl delete slurmcluster my-slurm-cluster -n slurm
```
:::warning
Deleting the `SlurmCluster` does not delete the shared `/home` volume. It must be manually deleted via the Crusoe Console or CLI.
:::
When the `SlurmCluster` is deleted, CSO automatically removes the controller and login node pools it created. You then need to delete the compute node pools (or repurpose them for other workloads) and finally the CMK cluster itself:
```sh
crusoe kubernetes nodepools delete slurm-h100-workers --cluster-name my-slurm-cluster
crusoe kubernetes clusters delete my-slurm-cluster
```
## Troubleshooting
### Cluster stuck in `Provisioning` or `Installing`
Look at the conditions:
```sh
kubectl describe slurmcluster my-slurm-cluster -n slurm
```
Common causes:
- **`LoginReady` / `ControllerReady` stuck**: CSO provisions these node pools automatically when you apply the SlurmCluster CR. If they're stuck, check that the underlying VM provisioning succeeded — `crusoe kubernetes nodepools list --cluster-name my-slurm-cluster` should show CSO-managed controller and login pools in `RUNNING` state. Quota or capacity issues at the project level are the most common reason.
- **`SlinkyReady` / `TopographReady` failing**: the operator's Helm install is failing. Check the operator pod logs: `kubectl logs -n slurm deploy/crusoe-slurm-operator`.
- **`CertManagerReady` failing**: cert-manager pods aren't healthy. Check `kubectl get pods -n cert-manager`.
### Compute nodes don't appear in `sinfo`
Verify the labels on your compute nodes:
```sh
kubectl get nodes -l slurm.crusoe.ai/compute-node-type=true \
-L crusoe.ai/nodepool.id,crusoe.ai/nodepool.name
```
Each compute node must have:
- `slurm.crusoe.ai/compute-node-type=true` (set by you on the node pool)
- `crusoe.ai/nodepool.id=` (auto-set by CMK)
If labels are missing, the operator's `NodeReconciler` will not create a NodeSet for them. Re-create the node pool with the correct `--node-labels` argument.
For a deeper diagnostic dump (recommended when escalating to support), run the slurm-debug tool — see the [Advanced: Kubernetes Operations](./advanced-kubernetes.md) guide.
## Next Steps
- [User Management](./user-management.md) — Add users and groups, manage sudo access
- [Managing Partitions](./managing-partitions.md) — Create custom Slurm partitions
- [Node Health Checks](./node-health-checks.md) — Built-in health checks and adding your own health/prolog/epilog checks
- [Slurm Metrics](./slurm-metrics.md) — Monitor cluster and job performance
- [Advanced: Kubernetes Operations](./advanced-kubernetes.md) — Direct CRD reference, AutoClusters / SIGTERM handling
---
# Slurm Metrics
Crusoe Managed Slurm metrics provide comprehensive insights into the performance and utilization of your Slurm clusters. These metrics help you monitor cluster health, job performance, resource utilization, and identify bottlenecks in your HPC workloads.
Crusoe Cloud collects Slurm-specific metrics alongside infrastructure metrics (GPU, CPU, memory, disk, and network) for your Slurm clusters. Metrics are collected in 60-second intervals and retained for 30 days.
The Crusoe Watch Agent automatically detects and scrapes Slurm metrics when a Slurm controller pod is present in the cluster. You can customize or disable this behavior through agent configuration.
## Prerequisites
:::note
Clusters created through the Slurm API (the Slurm tab on Crusoe Cloud console or the `crusoe slurm` CLI) have all of the following pre-requisites installed out of the box. You can disregard the pre-requisites below.
:::
To use Slurm Metrics, you need:
- A running Managed Slurm cluster (see [Quickstart](./quickstart.md))
- Crusoe Watch Agent installed on the CMK cluster (see [Getting Started](../../command-center/get-started.mdx), agent version 0.3.11 or later)
- NVIDIA GPU Operator add-on enabled (included by default with Managed Slurm)
## Configuring Slurm Metrics Collection
The Crusoe Watch Agent automatically scrapes Slurm metrics when it detects a Slurm controller pod in your cluster. You can customize this behavior by configuring the agent.
### Default Behavior
By default, the agent scrapes the following Slurm metrics endpoints:
- `/metrics/jobs` — Job-level metrics
- `/metrics/jobs-users-accts` — User and account job metrics
- `/metrics/nodes` — Node state and allocation metrics
- `/metrics/partitions` — Partition metrics
- `/metrics/scheduler` — Scheduler performance metrics
### Customizing Metrics Collection
To customize which metrics are collected, create a `values.yaml` file with your preferred Slurm metrics configuration:
```yaml
slurmMetrics:
enabled: true
paths:
- /metrics/jobs
- /metrics/jobs-users-accts
- /metrics/nodes
- /metrics/partitions
- /metrics/scheduler
```
### Disabling Slurm Metrics
To disable Slurm metrics collection entirely:
```yaml
slurmMetrics:
enabled: false
```
### Applying Configuration
Apply your custom configuration using Helm:
```sh
helm upgrade crusoe-watch-agent crusoe-watch-agent/crusoe-watch-agent --namespace crusoe-system -f values.yaml
```
## Available Metrics
Slurm metrics are accessible through the Prometheus-compatible query API.
### Slurm Job Metrics
- **Job queue length** — Number of jobs waiting in the queue
- **Running jobs** — Number of currently executing jobs
- **Job wait time** — Time jobs spend in queue before execution
- **Job completion rate** — Rate of job completions over time
### Slurm Node Metrics
- **Node state** — Current state of Slurm nodes (idle, allocated, down, drain)
- **Node allocation** — Percentage of nodes allocated vs. idle
- **Node availability** — Number of available nodes for job scheduling
### Resource Utilization Metrics
In addition to Slurm-specific metrics, you can access all standard infrastructure metrics for your Slurm cluster nodes:
- GPU utilization, memory, temperature, and power draw
- CPU utilization and system memory
- Network bandwidth (VPC and InfiniBand)
- Storage I/O metrics
For a complete list of infrastructure metrics, see [CMK Telemetry](../cmk/cmk-telemetry.md) and [VM Telemetry](../../compute/virtual-machines/vm-telemetry.md).
## Accessing Slurm Metrics
Slurm metrics are accessible via the API, Telemetry Conduit, and Crusoe MCP. They're not available in the Console. See [Getting Started](../../command-center/get-started.mdx) for instructions on generating a monitoring token and using each access method.
### Via API
Query Slurm metrics using the Prometheus-compatible API endpoint:
```sh
https://api.cloud.crusoe.ai/v1/projects//metrics/timeseries
```
**Example query for Slurm job queue length:**
```sh
curl -G https://api.cloud.crusoe.ai/v1/projects//metrics/timeseries\?query=\
slurm_queue_length{cluster_id=""} \
-H 'Authorization: Bearer '
```
### Grafana Dashboard
Pre-built Grafana dashboard templates for Managed Slurm clusters are available in the [Crusoe solutions library](https://github.com/crusoecloud/solutions-library/tree/main/grafana-cmk). Templates cover Slurm job performance, GPU utilization, InfiniBand fabric health, power draw, XID error tracking, storage, and network.
## Monitoring Best Practices
### Tracking Job Performance
Monitor job wait times and queue lengths to identify scheduling bottlenecks. High wait times may indicate:
- Insufficient compute resources — consider adding more node sets
- Inefficient job packing
- Need for additional node sets with different GPU types
### Resource Optimization
Use GPU and CPU utilization metrics alongside Slurm job metrics to:
- Identify underutilized nodes
- Optimize job resource requests
- Right-size node sets for your workload
### Cluster Health
Monitor node state metrics to detect:
- Nodes in drain state requiring attention
- Hardware failures affecting job scheduling
- Capacity constraints
## Next Steps
- [Quickstart](./quickstart.md) — Set up your Slurm cluster
- [User Management](./user-management.md) — Add users and groups to your cluster
- [Managing Partitions](./managing-partitions.md) — Create and manage partitions in your Slurm cluster
- [Node Health Checks](./node-health-checks.md) — Built-in health checks and adding your own health/prolog/epilog checks
- [Advanced: Kubernetes Operations](./advanced-kubernetes.md) — Direct kubectl access and CRD-level configuration
- For Slurm command reference, see the [official Slurm documentation](https://slurm.schedmd.com/)
---
# Advanced: Kubernetes Operations
Managed Slurm clusters run on Crusoe Managed Kubernetes (CMK). While the CLI and API handle most operations, you can also interact with the cluster directly via `kubectl` for advanced configuration, troubleshooting, and user management.
## Accessing Your Cluster via kubectl
Configure `kubectl` to connect to your cluster:
```sh
crusoe kubernetes clusters get-credentials
```
Verify the connection:
```sh
kubectl cluster-info
```
## Viewing Cluster State
### Slurm Custom Resources
The Crusoe Slurm Operator (CSO) uses four Custom Resource types to manage your cluster. When you create a Slurm cluster through the Slurm UI or `crusoe slurm` CLI, these CRDs will be created, controlled, and reconciled by Crusoe in the `slurm` namespace.
```sh
kubectl get slurmclusters -n slurm # Clusters
kubectl get nodesets -n slurm # Node sets
kubectl get slurmusers -n slurm # All users
kubectl get slurmusergroups -n slurm # All groups
```
For detailed status with conditions:
```sh
kubectl describe slurmclusters -n slurm
```
:::tip
Use `kubectl explain` for field-level documentation on any Custom Resource. The CRDs have built-in descriptions for every field:
```sh
kubectl explain slurmclusters.spec
kubectl explain slurmusers.spec
kubectl explain slurmusergroups.spec
kubectl explain nodesets.spec
```
:::
### Cluster Phase and Conditions
The SlurmCluster status includes a `phase` field (possible values: `Provisioning`, `Installing`, `Ready`, `NotReady`, `CreateFailed`, `Deleting`, `DeleteFailed`) and detailed conditions for each component. Use `kubectl describe` to see what's happening if your cluster isn't healthy:
```sh
kubectl describe slurmclusters -n slurm
```
Look for conditions like:
- `ControllerReady` — Slurm controller pod is running
- `LoginReady` — Login nodes are running
- `SlinkyReady` — Slinky Helm chart is installed
- `CertManagerReady` — cert-manager is installed
- `LoadBalancerReady` — Load balancer controller is installed
- `TopographReady` — Topology discovery is running
### Node Set Readiness
The NodeSet status shows a `readyReplicas` field in "ready/total" format:
```sh
kubectl get nodesets -n slurm
```
```
NAME READY AGE
slurm-worker-node-set 2/2 1h
```
## Storage Configuration
Managed Slurm uses a shared filesystem for the `/home` directory, mounted across all login and worker nodes. This is backed by a PersistentVolumeClaim (PVC) using the Crusoe CSI driver.
### StorageClass
The operator automatically creates a StorageClass named `-crusoe-csi-driver-fs-sc` with:
- **Provisioner:** `fs.csi.crusoe.ai`
- **Volume binding mode:** WaitForFirstConsumer
- **Volume expansion:** Enabled
### Viewing Storage
```sh
kubectl get pvc -n slurm # View persistent volume claims
kubectl get sc # View storage classes
```
## What the Operator Manages
The Crusoe Slurm Operator (CSO) continuously reconciles certain resources to keep your cluster in a healthy state. Understanding what CSO manages helps you know what's safe to modify and what will be reverted.
### Reconciliation Reference
| Resource | Managed by CSO? | Safe to Modify? | Notes |
| ----------------------------------------------- | -------------------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------- |
| **SlurmUser CRs** | No — customer-owned | Yes | This is the intended way to manage users |
| **SlurmUserGroup CRs** | No — customer-owned | Yes | This is the intended way to manage groups |
| **SlurmClusterHealthCheck** (`-custom`) | Created once — never overwritten | Yes | Add your own health/prolog/epilog checks here (see [Node Health Checks](./node-health-checks.md)) |
| **SlurmClusterHealthCheck** (`-defaults`) | Yes — seeded and owned by CSO | No | Crusoe-provided node health checks |
| **Slinky CRDs** (Controller, LoginSet, NodeSet) | Yes — full overwrite on every reconcile | No | Changes will be reverted automatically |
| **gres-config ConfigMap** | Seeded once — your edits are kept | Yes | Sets the GPU detection backend. Defaults to `AutoDetect=nvml` |
| **cgroup-config ConfigMap** | Seeded once — your edits are kept | Yes | Holds `cgroup.conf`. An admission policy rejects edits that disable the cgroup plugin |
| **plugstack-config ConfigMap** | Yes — reconciled every cycle | No | Loads the Pyxis SPANK plugin via `plugstack.conf` |
| **Auth Secrets** (slurm-auth, jwt-auth) | Create-once — never regenerated | Do not modify | Breaking these breaks cluster authentication |
| **nsscache ConfigMap** | Yes — regenerated on user changes | No | Auto-managed from SlurmUser CRs |
| **ssh-keys Secret** | Yes — regenerated on user changes | No | Auto-managed from SlurmUser CRs |
| **topology.conf ConfigMap** | Created by CSO, then managed by Topograph | No | Topograph updates this automatically based on network topology |
| **Home PVC** | Validated but not overwritten after creation | Storage size only (increase) | Spec is immutable after initial creation |
| **StorageClasses** | Yes — reconciled every cycle | No | Provisioner and settings are fixed |
| **Your own resources** | Never | Yes | CSO ignores any resources it doesn't own |
A ConfigMap marked _seeded once_ keeps your edits. To return one to the Crusoe default, delete its key. CSO restores the default file on the next reconcile.
:::info
**Key takeaway:** You edit the `SlurmUser` and `SlurmUserGroup` CRs (see [User Management](./user-management.md)), the `SlurmClusterHealthCheck` `-custom` object (see [Node Health Checks](./node-health-checks.md)), and the ConfigMaps marked _seeded once_. Everything else is either managed by CSO or managed via the Crusoe CLI/API.
:::
## Slurm Commands Reference
Once connected to a login node via SSH, use standard Slurm commands:
| Command | Description |
| ---------------------- | ---------------------------------------- |
| `sinfo` | View cluster status and node information |
| `sinfo -R` | View nodes in drain state with reasons |
| `squeue` | View the job queue |
| `sbatch` | Submit a batch job |
| `srun` | Run a job interactively |
| `scancel` | Cancel a job |
| `scontrol show node` | View detailed node information |
| `scontrol show config` | View Slurm configuration |
## Running NCCL Tests
To validate multi-node GPU communication, run an NCCL all-reduce test. SSH into a login node and create the following script named `nccl_test.batch`. This example runs NCCL tests on H200 nodes. Different hardware types will have their own topo files. Note that you can find more test examples on your login node at `/opt/examples`:
```sh
#!/bin/bash
#SBATCH --job-name=nccl_tests
#SBATCH --nodes=
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=20:00
#SBATCH --output="%x_%j.out"
#SBATCH --exclusive
export NCCL_TOPO_FILE=/etc/crusoe/nccl_topo/h200-141gb-sxm-ib-cloud-hypervisor.xml
export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_HCA="mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1"
export NCCL_IB_MERGE_VFS=0
export NCCL_DEBUG=WARN
export OMPI_MCA_coll_hcoll_enable=0
export PMIX_MCA_gds='^ds12'
export UCX_NET_DEVICES="mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1"
srun --mpi=pmix /opt/nccl-tests/build/all_reduce_perf -b 2G -e 32G -f 2
```
:::note
Update the `NCCL_TOPO_FILE` path to match your GPU type. The example above is for H200 nodes.
:::
Submit the test:
```sh
sbatch nccl_test.batch
```
Monitor with `squeue`, and check the output file `nccl_tests_.out` once complete.
To connect interactively to a worker node:
```sh
srun --pty bash # Any available worker
srun --nodelist= --pty bash # A specific worker
```
## Health Checks, Prolog, and Epilog
Managed Slurm ships a built-in node health-check suite (periodic, prolog, and epilog) and lets you add your own health, prolog, and epilog checks through the `SlurmClusterHealthCheck` custom resource. See [Node Health Checks](./node-health-checks.md).
## Automatic Hardware Remediation
Managed Slurm clusters run on Crusoe Managed Kubernetes with [AutoClusters](../cmk/autoclusters.md) enabled. AutoClusters automatically detects critical hardware failures — such as a GPU or HCA falling off the bus — and remediates them without manual intervention.
### What Happens During Remediation
1. Crusoe's monitoring pipeline detects a hardware issue on a worker node
2. The affected Slurm node is set to **DOWN**, which immediately cancels any running job on that node — the job process receives a **SIGTERM** signal before being terminated
3. You have up to **2 minutes** to handle the SIGTERM (save checkpoints, flush logs, etc.) before the node is replaced
4. The cancelled job is automatically requeued (`JobRequeue=1` is enabled by default) and runs on a healthy node
### Handling SIGTERM in Your Jobs
When a node goes down, Slurm sends SIGTERM to your job process. You can use `trap` to catch this signal and perform cleanup before the job is cancelled. For example:
```sh
#!/bin/bash
#SBATCH --job-name=my-training-job
#SBATCH --nodes=1
#SBATCH --output=%x-%j.out
trap 'echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] SIGTERM received — saving checkpoint"; save_checkpoint' SIGTERM
run_training &
wait $!
```
## Next Steps
- [Quickstart](./quickstart.md) — Create your first Slurm cluster
- [User Management](./user-management.md) — Add users and groups, manage partitions
- [Managing Partitions](./managing-partitions.md) — Create and manage partitions in your Slurm cluster
- [Node Health Checks](./node-health-checks.md) — Built-in health checks and adding your own health/prolog/epilog checks
- [Slurm Metrics](./slurm-metrics.md) — Monitor cluster health and performance
- For Slurm command reference, see the [official Slurm documentation](https://slurm.schedmd.com/)
## Support
If you encounter issues or need assistance, contact [Crusoe Cloud Support](https://support.crusoecloud.com/).
---
# Overview
The Crusoe Container Registry (CCR) is a Docker-compliant container registry that allows you to store container images in a specific Crusoe location. This improves latency when scaling workloads.
CCR supports two modes:
- **Standard:** A private, read/write repository hosted in Crusoe for your container images.
- **Pull-Through Cache:** A private, read-only cache for an upstream public or private registry. Images are fetched from your upstream source, cached in your CCR repository, and served to your machine.
## Key Benefits
- **Reduce Latency:** Store images in the same Crusoe location as your compute to accelerate workload startup times.
- **Eliminate Egress Costs:** Avoid data egress charges from external cloud registries when scaling clusters.
- **Improve Reliability:** Mitigate `ImagePullBackOff` errors in Kubernetes caused by rate-limiting from public registries, especially when many nodes pull the same image simultaneously.
## Pricing
**Starting on Thursday, January 15th, 2026,** any usage of CCR incurs charges at a rate of $0.10/GiB/month.
## Best Practices and Limitations
- **Region Co-location:** To ensure high throughput and low latency, provision your repositories in the same location as your compute clusters.
---
# Quickstart with Docker
# Quickstart: Pull an Image with Docker
### Step 1: Create a Repository
A repository is an abstract folder you create within CCR, often so you can organize your images per team, business function, or deployment environment.
**CLI:**
```sh
crusoe registry repositories create \
--name my-nginx-repo \
--location us-east1-a \
--mode standard
```
Your repository URL will be returned. CCR repository URLs follow a standard format of:
`registry..ccr.crusoecloudcompute.com/.`.
**UI:**
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Click **Create Repository**.
3. Select **Standard** mode.
4. Click **Create**.
Your repository URL will be in the header of your repository details page. CCR repository URLs follow a standard format of:
`registry..ccr.crusoecloudcompute.com/.`.
### Step 2: Create an Authentication Token
Create a token to authenticate the Docker client to your registry.
**CLI:**
```sh
crusoe registry tokens create --alias my-quickstart-token
```
**UI:**
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Select your repository.
3. Click **Create Token**.
Save the generated token in a secure location; _it will not be shown again_.
### Step 3: Log in with Docker
Use the `docker login` command to authenticate to your registry from your machine. Use your Crusoe account email address as the username, and the token you just created as the password when prompted.
```sh
docker login registry.us-east1-a.ccr.crusoecloudcompute.com/my-nginx-repo.7hf6et43 \
-u your-email@example.com \
-p 'my-crusoe-registry-token'
```
:::info
Docker registry tokens often contain special characters (like $) that your shell may try to interpret as variables. To prevent this, always enclose your token in single quotes (e.g., --password 'my-crusoe-registry-token').
:::
### Step 4: Tag and Push an Image
The following example pulls a standard image from Docker Hub, re-tags it for a CCR repository, and pushes it.
```sh
docker pull nginx:latest
docker tag nginx:latest registry.us-east1-a.ccr.crusoecloudcompute.com/my-nginx-repo.7hf6et43/nginx:latest
docker push registry.us-east1-a.ccr.crusoecloudcompute.com/my-nginx-repo.7hf6et43/nginx:latest
```
### Step 5: Verify the Push
List the images in your repository to confirm the push was successful.
**CLI:**
```sh
crusoe registry images list my-nginx-repo --location us-east1-a
```
**UI:**
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Select your repository.
3. Make sure your image is listed.
---
# Managing Repositories
A repository is an abstract folder you create within CCR, often so you can organize your images per team, business function, or deployment environment.
### Creating a Standard Repository
**CLI:**
```sh
crusoe registry repositories create \
--name my-nginx-repo \
--location us-east1-a \
--mode standard
```
**UI:**
To create a standard repository using the [console](https://console.crusoecloud.com):
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Click **Create Repository**.
3. Select **Standard** mode.
4. Click **Create**.
Your repository URL will be in the header details at the top of the page.
### Creating a Pull-Through Cache
A read-only cache for an upstream registry. When you pull an image, it is fetched from the upstream source, stored in your CCR repository, and then served.
#### Configuring Upstream Providers
When you create a repository in pull-through cache mode, you must specify an upstream registry `provider` and `url`, and if applicable, a `username` and `password`. Below are instructions for common upstream providers. Any Docker V2-compliant registries not explicitly listed should use `provider: docker-registry`.
| Registry Provider | Example URL (must start with `http` or `https`) | `provider` | `username` | `password` |
| ------------------ | :---------------------------------------------- | :---------------- | :------------------------------------------------ | :------------------------------------ |
| Google Cloud (GAR) | `https://-docker.pkg.dev` | `google-gar` | `_json_key` | Entire service account JSON key file |
| Google Cloud (GCR) | `https://gcr.io` | `google-gcr` | `_json_key` | Entire service account JSON key file |
| AWS ECR (Private) | `https://.dkr.ecr..amazonaws.com` | `aws-ecr` | IAM user's access key ID | IAM user's secret key |
| AWS ECR Public | `https://public.ecr.aws` | `docker-registry` | N/A | N/A |
| Docker Hub | `https://hub.docker.com` | `docker-hub` | `` | `` |
| GitHub (GHCR) | `https://ghcr.io` | `github-ghcr` | `` | `` |
| Azure (ACR) | `https://.azurecr.io` | `azure-acr` | `` or `` | `` or `` |
| GitLab | `https://registry.gitlab.com` | `docker-registry` | `` | `` |
| NVIDIA NGC | `https://nvcr.io` | `docker-registry` | `$oauthtoken` | NGC API key |
| Oracle (OCI) | `https://.ocir.io` | `docker-registry` | `/` | OCI Registry Token |
**CLI:**
```sh
crusoe registry repositories create \
--name my-dockerhub-cache \
--location us-east1-a \
--mode pull-through-cache \
--upstream-registry \
url=https://hub.docker.com,provider=docker-hub,username=myuser,password=mypass
```
To see all supported upstream registry providers, use `crusoe registry supported providers`.
**UI:**
To create a pull-through cache repository using the [console](https://console.crusoecloud.com):
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Click **Create Repository**.
3. Select **Pull-through Cache** mode.
4. Select your upstream registry provider.
5. Enter your upstream registry URL and, optionally, your credentials.
6. Click **Create**.
Your repository URL will be in the header details at the top of the page.
### Listing and Getting Repository Details
**CLI:**
**List repositories in your project:**
```sh
crusoe registry repositories list
```
**Get details, including URL, for a specific repository:**
```sh
crusoe registry repositories get my-app-repo --location us-east1-a
```
**UI:**
To view all repositories in the [console](https://console.crusoecloud.com/registry), go to the [Container Registry](https://console.crusoecloud.com/registry) page.
### Deleting a Repository
A repository must be empty to be deleted. To delete images, see [managing images](managing-images.md#deleting-images).
**CLI:**
```sh
crusoe registry repositories delete my-app-repo --location us-east1-a
```
**UI:**
To delete a repository using the [console](https://console.crusoecloud.com):
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Find the repository you want to delete.
3. Click the **Delete** icon next to the repository you want to delete.
### Repository URL Structure
CCR repository URLs follow a standard format of:
`registry..ccr.crusoecloudcompute.com/.`
---
# Managing Images
An image, such as `nginx`, can have multiple manifests, which are unique versions or builds of nginx. A specific manifest can be referenced by using a tag (e.g. `nginx:latest`, `nginx:1.18`, or `nginx:production`), which is a mutable pointer, or using its digest (e.g. `nginx@sha256:94a1...`), which is an immutable hash of its contents. Each manifest has one unique digest but can have many tags pointing to it.
### Listing Images
**CLI:**
```sh
crusoe registry images list my-repo --location us-east1-a
```
**UI:**
To view a repository's images in the [console](https://console.crusoecloud.com/registry):
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Select a repository.
### Listing Manifests
**CLI:**
```sh
crusoe registry manifests list nginx --repo-name my-repo --location us-east1-a
```
**UI:**
To view an image's manifests in the [console](https://console.crusoecloud.com):
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Select the repository containing your image.
3. Select the image to view its details page including all manifests by tag/digest.
### Deleting Images
When you delete an image, all of its tags and manifests are deleted.
**CLI:**
```sh
crusoe registry images delete my-image-name --repo-name my-repo --location us-east1-a
```
**UI:**
To delete an image using the [console](https://console.crusoecloud.com):
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Click on your repository to see all of its details and images.
3. Select the image you want to delete. You can delete images from the repository details page or the image details page.
### Deleting Manifests
You can delete specific manifests by tag or digest.
**CLI:**
Deleting a manifest by digest
```sh
crusoe registry manifests delete --repo-name \
--location \
--digest
```
Deleting a manifest by tag
```sh
crusoe registry manifests delete --repo-name \
--location \
--tag
```
**UI:**
To delete a manifest by tag or digest using the [console](https://console.crusoecloud.com):
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Select the repository containing your image.
3. Select the image.
4. From the image details page, select the **Delete** icon next to the tags/digests you want to delete.
---
# Pulling Images into a Kubernetes Cluster
To pull images from a CCR repository into your Kubernetes cluster, such as a Crusoe Managed Kubernetes (CMK) cluster, you must provide credentials via a Kubernetes `Secret`. This allows your pods to authenticate with CCR.
### Step 1: Create a Registry Token
First, generate a new, long-lived token for your cluster to use.
**CLI:**
```sh
crusoe registry tokens create --alias prod-cluster-token
```
**UI:**
1. Go to the [Container Registry](https://console.crusoecloud.com/registry) page.
2. Select your repository.
3. Click **Create Token**.
Save the generated token in a secure location; _it will not be shown again_.
### Step 2: Create the Kubernetes Secret
Next, use `kubectl` to create a `docker-registry` secret in your cluster. Provide your CCR repository URL, your Crusoe account email as the username, and the token from the previous step as the password.
```sh
kubectl create secret docker-registry ccr-credentials \
--docker-server= \
--docker-username= \
--docker-password='' \
--namespace=my-app-namespace
```
:::info
Docker registry passwords often contain special characters (such as $) that are interpreted as variables by your command line shell. Enclose your token in single quotes (') when using the CLI, as shown above, to avoid errors during secret creation and image pulls.
:::
### Step 3: Reference the Secret in a Deployment
In your Kubernetes Deployment manifest, reference the secret in the `spec.template.spec.imagePullSecrets` field. This allows pods created by this deployment to authenticate with CCR.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-app-on-kubernetes
spec:
template:
spec:
containers:
- image: registry.us-east1-a.ccr.crusoecloudcompute.com/my-app-repo.7dhg29ls/my-app:v1.2
name: app-image
imagePullSecrets:
- name: ccr-credentials
```
Now, when you apply this deployment, your pods will be able to successfully pull the private image from your CCR repository.
---
# CCR Token Rotation in Kubernetes
# Rotating Registry Tokens in Kubernetes
By default, CCR tokens are static and short-lived. We recommend rotating tokens frequently. The Crusoe Token Rotator Helm chart runs a periodic job in your cluster to refresh the tokens and update your Kubernetes Secret. The source code and chart can be found in the [crusoe-registry-token-rotator-helm-charts](https://github.com/crusoecloud/crusoe-registry-token-rotator-helm-charts) repository.
## Overview
The Token Rotator runs as a Kubernetes `CronJob`. On a defined schedule, it:
1. Uses your CMK cluster's Crusoe API credentials to generate a new short-lived registry token.
2. Updates the `crusoe-image-pull-secrets` in your cluster with the new token.
3. Cleans up any old tokens managed by the rotator.
This ensures that your Kubernetes nodes always have valid credentials to pull images without manual intervention.
## Prerequisites
Before installing the rotator, you will need:
- An existing **CCR Repository**.
- A Kubernetes cluster with `helm >= 3.x` and `kubectl >= 1.21.x` installed.
### Create Namespace and Secret (Non-CMK Clusters Only)
If you are not using Crusoe Managed Kubernetes (CMK), you will need Crusoe API keys. You can create them by following the instructions in ["Manage your API Keys"](../identity-and-security/managing-api-keys.mdx).
You must then create both the `crusoe-system` namespace and the `crusoe-secrets` secret:
```sh
kubectl create namespace crusoe-system
kubectl create secret generic crusoe-secrets \
--from-literal=CRUSOE_ACCESS_KEY= \
--from-literal=CRUSOE_SECRET_KEY= \
-n crusoe-system
```
## Installation
### Step 1: Download the Rotator
First, ensure your `kubectl` context is set to the correct cluster:
```sh
kubectl config current-context
```
Now clone the repository with the Token Rotator Helm chart:
```sh
git clone https://github.com/crusoecloud/crusoe-registry-token-rotator-helm-charts.git
cd crusoe-registry-token-rotator-helm-charts
```
### Step 2: Configure the Rotator
Update at least the following fields in `charts/crusoe-registry-token-rotator/values.yaml`:
- `targetSecret.registryUrl` (required): Set to your CCR repository URL
- `targetSecret.registryUsername` (required): Set to your CCR username, which is the email address you use in Crusoe
- `targetSecret.namespaces` (optional): Update if you want the Secret created in namespaces other than `default`
For example:
```yaml
image:
repository: ghcr.io/crusoecloud/crusoe-registry-token-rotator
tag: "latest"
pullPolicy: IfNotPresent
targetSecret:
name: crusoe-image-pull-secrets
namespaces:
- default
# -
# -
registryUrl: ""
registryUsername: "