VM Telemetry
Command Center provides you visibility into the health and performance of your Crusoe virtual machines. Two categories of telemetry are available for VMs:
Metrics: Infrastructure metrics covering GPU, CPU, memory, disk, and network performance, collected at 60-second intervals and retained for 30 days. Available for NVIDIA GPU, AMD GPU, and non-GPU VMs. A subset is viewable in the Console; the full dataset is available via Prometheus-compatible API and Grafana.
In the Console, navigate to Compute, select your VM, then select the Metrics tab.
Logs: JournalD system logs are collected from each VM and available to search, filter, and query. Supported for NVIDIA GPU accelerated instances, AMD GPU accelerated instances, and non-GPU instances. Logs are retained for 7 days. In the Console, navigate to Managed Logs in the left navigation bar to search across all VMs, or navigate to Compute, select your VM, then select the Logs tab.
Custom metrics (application-defined metrics from your workloads) are available for CMK clusters only. You can't use custom metrics for standalone VMs.
For installation, token generation, and access method details, see Get started. VM telemetry requires agent version vm-v1.0.3 or higher.
VM Instance Actions
After you install the Crusoe Watch Agent, you can perform the following actions from your VM instance page in the Crusoe Console:
- Generate bug report — Create an NVIDIA or AMD bug report for VMs with NVIDIA or AMD GPUs. Download it or attach it to a support ticket for troubleshooting GPU-related issues. For possible error messages during generation, see Bug Report Generation Error Messages.
- Report an issue — Open a pre-filled Zendesk support ticket with VM information and attach the latest bug report if applicable.
To access these features:
- Navigate to Compute in the left navigation bar.
- Select your VM from the list.
- Click on the three vertical dots icon next to the Start/Stop VM button and then click Generate bug report or Report issues.
NVIDIA bug report generation requires agent version vm-v1.0.3 or above and is available for VMs with NVIDIA GPUs. AMD bug report generation requires agent version vm-v1.0.5 or above and is available for VMs with AMD GPUs.
NVIDIA GPU Metrics
The following metrics are available for VMs with NVIDIA GPU instances:
| Metrics | Definition | Suggested Query |
|---|---|---|
| TFlops (FP16) | The measured 16-bit floating-point GPU throughput calculated by scaling the tensor core utilization against the hardware's theoretical maximum. | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE / 100 * theoretical max TFLOPS |
| GPU Utilization (%) | The percentage of time the GPU is actively executing tasks. | DCGM_FI_DEV_GPU_UTIL |
| CPU Utilization (%) | The aggregated percentage of time the host's CPU cores are busy over the last 60 seconds. | (sum without(cpu, mode) (rate(crusoe_vm_cpu_seconds_total{vm_id="vm-id", mode!="idle"}[60s]))) / (sum without(cpu, mode) (rate(crusoe_vm_cpu_seconds_total{vm_id="vm-id"}[60s]))) * 100 |
| GPU Memory Utilization (%) | The percentage of total dedicated GPU memory that is actively allocated and consumed by processes on the GPU. | (DCGM_FI_DEV_FB_USED / ( DCGM_FI_DEV_FB_FREE + DCGM_FI_DEV_FB_USED)) * 100 |
| GPU Memory Bandwidth Utilization (%) | The percentage of the theoretical peak memory interface bandwidth being utilized for data transfer between the GPU and memory. | DCGM_FI_PROF_DRAM_ACTIVE |
| System Memory Utilization (%) | The percentage of total host system RAM that is consumed. | (crusoe_vm_memory_used_bytes / crusoe_vm_memory_total_bytes) * 100 |
| GPU Power Draw (W) | The current power consumption of the GPU, measured in Watts. | DCGM_FI_DEV_POWER_USAGE |
| GPU Temperature (Celsius) | The current core temperature of the GPU die, measured in Celsius. | DCGM_FI_DEV_GPU_TEMP |
| Tensor Core Utilization (%) | The percentage of time the Tensor pipeline is actively processing instructions over the sample period. | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE * 100 |
| VPC Network Bandwidth In (bytes per second) | The rate of data received by the host machine via the VPC network interface, measured in bytes per second. | crusoe_vm_network_receive_bytes_total |
| VPC Network Bandwidth Out (bytes per second) | The rate of data transmitted by the host machine via the VPC network interface, measured in bytes per second. | crusoe_vm_network_transmit_bytes_total |
| PCIe Bandwidth (bytes per second) | The rate of data transfer (Tx + Rx) between the CPU host memory and the GPU over the PCIe bus, measured in bytes per second. | DCGM_FI_PROF_PCIE_TX_BYTES + DCGM_FI_PROF_PCIE_RX_BYTES |
| PCIe Replay Rate | The rate of error-induced packet retransmissions over the PCIe bus, measured in replays per second. High rates indicate link quality issues. | rate(DCGM_FI_DEV_PCIE_REPLAY_COUNTER[1m]) |
| Uncorrectable ECC Error Rate | The rate of accumulation of uncorrectable double-bit memory errors (DBE) on the GPU, indicating severe hardware instability. | rate(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1m]) |
| Correctable ECC Error Rate | The rate of accumulation of correctable single-bit memory errors (SBE) on the GPU, indicating marginal hardware stability. | rate(DCGM_FI_DEV_ECC_SBE_VOL_TOTAL[1m]) |
| SM Occupancy (%) | The average percentage of available resident warps running concurrently on the Streaming Multiprocessors (SMs) over the sample period. | DCGM_FI_PROF_SM_OCCUPANCY |
| SM Active (%) | The percentage of time the Streaming Multiprocessors (SMs) were executing instructions during the sample period. | DCGM_FI_PROF_SM_ACTIVE * 100 |
| SM Average Clock Speed (MHz) | The current instantaneous clock frequency of the GPU's Streaming Multiprocessors (SMs) in Megahertz (MHz). | DCGM_FI_DEV_SM_CLOCK |
| GPU XID error | The most recent unique error code emitted by the GPU driver (a hardware or software fault ID). Non-zero values indicate an error that typically requires a driver reset or GPU restart. | DCGM_FI_DEV_XID_ERRORS |
NVIDIA NVLink Metrics
If your VM uses NVLink-enabled NVIDIA instances, the following additional NVLink metrics are available:
| Metrics | Definition | Suggested Query |
|---|---|---|
| GPU NVLink Bandwidth In (bytes per second) | The rate of data received by the GPU from other GPUs over all active NVLink connections, measured in bytes per second. | DCGM_FI_DEV_NVLINK_RX_BYTES |
| GPU NVLink Bandwidth Out (bytes per second) | The rate of data transmitted by the GPU to other GPUs over all active NVLink connections, measured in bytes per second. | DCGM_FI_DEV_NVLINK_TX_BYTES |
AMD GPU Metrics
AMD GPU Metrics is currently in preview and available for AMD MI300x and MI355x instances. Please reach out to Crusoe Cloud Support to learn more.
If your VM uses AMD GPU instances, the following AMD-specific metrics are supported:
| Metrics | Definition | Suggested Query |
|---|---|---|
| GPU Memory Utilization (%) | The percentage of total dedicated GPU memory that is actively allocated and consumed by processes on the GPU. | (gpu_used_visible_vram / gpu_total_visible_vram) * 100 |
| GPU Utilization (%) | The percentage of time the GPU graphics engine is actively executing tasks. | gpu_gfx_activity |
| GPU Power Draw (W) | The current power consumption of the GPU, measured in Watts. Requires AMD Device Metrics Exporter v1.5.0 or later. | gpu_power_usage |
| GPU Memory Bandwidth Utilization (%) | The percentage of the theoretical peak memory interface bandwidth being utilized for data transfer between the GPU and memory. | gpu_umc_activity |
| PCIe Bandwidth (bytes per second) | The rate of data transfer (Tx + Rx) between the CPU host memory and the GPU over the PCIe bus, measured in bytes per second. | pcie_bandwidth |
| GPU Temperature (Celsius) | The current junction temperature of the GPU, measured in Celsius. | gpu_junction_temperature |
| GPU XGMI Bandwidth In (bytes per second) | The rate of data received by the GPU from other GPUs over XGMI (AMD's high-speed interconnect), measured in bytes per second over the last minute. | rate(gpu_xgmi_link_rx[1m]) |
| GPU XGMI Bandwidth Out (bytes per second) | The rate of data transmitted by the GPU to other GPUs over XGMI (AMD's high-speed interconnect), measured in bytes per second over the last minute. | rate(gpu_xgmi_link_tx[1m]) |
| PCIe Replay Rate (replays per second) | The rate of error-induced packet retransmissions over the PCIe bus, measured in replays per second over the last minute. High rates indicate link quality issues. | rate(pcie_replay_count[1m]) |
| Uncorrectable ECC Error Rate | The rate of accumulation of uncorrectable ECC memory errors on the GPU over the last minute, indicating severe hardware instability. | rate(gpu_ecc_uncorrect_total[1m]) |
| Correctable ECC Error Rate | The rate of accumulation of correctable ECC memory errors on the GPU over the last minute, indicating marginal hardware stability. | rate(gpu_ecc_correct_total[1m]) |
JournalD system logs are also collected for AMD GPU accelerated instances, the same as for NVIDIA GPU instances. Access them via Managed Logs in the Console or API.
CPU-Only Instance Metrics
The following metrics are available for CPU-only VMs (non-GPU instances):
| Metrics | Definition | Suggested Query |
|---|---|---|
| CPU Utilization (%) | The aggregated percentage of time the host's CPU cores are busy over the last 60 seconds. | (sum without(cpu, mode) (rate(crusoe_vm_cpu_seconds_total{vm_id="vm-id", mode!="idle"}[60s]))) / (sum without(cpu, mode) (rate(crusoe_vm_cpu_seconds_total{vm_id="vm-id"}[60s]))) * 100 |
| System Memory Utilization (%) | The percentage of total host system RAM that is consumed. | (crusoe_vm_memory_used_bytes / crusoe_vm_memory_total_bytes) * 100 |
| VPC Network Bandwidth In (bytes per second) | The rate of data received by the host machine via the VPC network interface, measured in bytes per second. | crusoe_vm_network_receive_bytes_total |
| VPC Network Bandwidth Out (bytes per second) | The rate of data transmitted by the host machine via the VPC network interface, measured in bytes per second. | crusoe_vm_network_transmit_bytes_total |
CPU, System Memory, and VPC Network metrics are also available for GPU-accelerated instances (both NVIDIA and AMD) in addition to their GPU-specific metrics.
Disk Metrics
The following metrics reflect boot disk and persistent disk activity, as observed from the VM. Ephemeral disk activity isn't included; see NVMe Drive Health Metrics below for ephemeral disk health metrics.
| Metrics | Definition | Suggested Query |
|---|---|---|
| Disk Read IOPS | The rate of disk read operations completed per second. | rate(crusoe_vm_disk_reads_completed_total[60s]) |
| Disk Write IOPS | The rate of disk write operations completed per second. | rate(crusoe_vm_disk_writes_completed_total[60s]) |
| Disk Read Bandwidth (bytes per second) | The rate of data read from disk. | rate(crusoe_vm_disk_read_bytes_total[60s]) |
| Disk Write Bandwidth (bytes per second) | The rate of data written to disk. | rate(crusoe_vm_disk_write_bytes_total[60s]) |
| Disk Read Latency (p99) | The 99th percentile latency of disk read operations. | histogram_quantile(0.99, rate(crusoe_vm_disk_read_latency_seconds_bucket[60s])) |
| Disk Write Latency (p99) | The 99th percentile latency of disk write operations. | histogram_quantile(0.99, rate(crusoe_vm_disk_write_latency_seconds_bucket[60s])) |
| Disk Space Used (%) | The percentage of total disk capacity currently in use, per mounted filesystem. | (crusoe_vm_disk_bytes_used / crusoe_vm_disk_bytes_total) * 100 |
| Disk Inode Usage (%) | The percentage of total inodes currently in use, per mounted filesystem. | (crusoe_vm_disk_inodes_used / crusoe_vm_disk_inodes_total) * 100 |
NVMe Drive Health Metrics
For VMs with Ephemeral Disks (local NVMe drives), the following SMART/health metrics are available:
NVMe drive health metrics are supported for s1a instances and all GPU instances except L40S.
| Metrics | Definition | Suggested Query |
|---|---|---|
| NVMe Critical Warning | Indicates whether a SMART critical warning bit is set (spare capacity low, temperature, reliability, read-only, volatile memory backup failed, or PMR unreliable). | crusoe_vm_nvme_smart_critical_warning == 1 |
| NVMe Media Errors | The cumulative count of uncorrectable media and data integrity errors. | rate(crusoe_vm_nvme_media_errors_total[1h]) |
| NVMe Percentage Used | The percentage of the drive's rated endurance consumed (100 indicates rated endurance has been reached). | crusoe_vm_nvme_percentage_used >= 90 |
| NVMe Available Spare (%) | The remaining spare capacity available on the drive. | crusoe_vm_nvme_available_spare |
| NVMe Power-On Hours | The lifetime number of hours the drive has been powered on. | crusoe_vm_nvme_power_on_hours |
Object Storage VM Metrics
The following metrics track this VM's connections to Object Storage endpoints:
| Metrics | Definition | Suggested Query |
|---|---|---|
| Object Store Connections | Total connection phases observed to the Object Storage endpoint. | rate(crusoe_vm_objectstore_connections_total[5m]) |
| Object Store Connection Latency | Average connection-phase latency to the Object Storage endpoint. | rate(crusoe_vm_objectstore_connection_latency_seconds[5m]) / rate(crusoe_vm_objectstore_connections_total[5m]) |
| Object Store TCP Retransmits | TCP retransmissions to the Object Storage endpoint. | rate(crusoe_vm_objectstore_tcp_retransmits_total[5m]) |
| Object Store Bytes Sent | Total bytes sent to the Object Storage endpoint. | rate(crusoe_vm_objectstore_bytes_sent_total[5m]) |
| Object Store Bytes Received | Total bytes received from the Object Storage endpoint. | rate(crusoe_vm_objectstore_bytes_recv_total[5m]) |
These metrics report aggregate connection-phase statistics per endpoint rather than per-request (GET/PUT) latency. TLS prevents distinguishing individual requests at the TCP connection layer, so per-request latency requires a proxy-based measurement approach instead.
These are VM-side metrics only; Object Storage doesn't currently expose bucket-level metrics. See Object Storage Overview for more on Object Storage itself.
Disk, NVMe drive health, and Object Storage VM metrics are currently available only via the API, Telemetry Conduit, and Crusoe MCP. Console support is planned for a future release.
For token generation and querying metrics via API or Grafana, see Get started and Metrics.
Considerations
Correctable ECC errors not emitted correctly for multi-GPU VMs and clusters
Correctable ECC errors (DCGM_FI_DEV_ECC_SBE_VOL_TOTAL) may not be emitted with all timeseries for multi-GPU VMs and clusters by the DCGM exporter due to a known NVIDIA NVLink metrics bug (GitHub Issue). We are working on a long-term fix. In the interim, you can manually force DCGM to start monitoring this field by running the following command within the VM:
dcgmi dmon -e 310
Clusters with Slurm images not retrieving metrics correctly due to pre-installed dcgm-exporter
If you have a pre-installed dcgm-exporter systemd service, it could conflict with the dcgm-exporter that would be installed as part of installing the Crusoe Watch Agent, causing metrics collection failures. To prevent this issue, use --replace-dcgm-exporter to replace your existing dcgm-exporter with the Crusoe version for full metrics collection. The service_name is an optional field that defaults to dcgm-exporter.service.
sudo crusoe-watch-agent --replace-dcgm-exporter [SERVICE_NAME]