Skip to main content

Diagnostics

Diagnostics let you capture the state of an instance at a point in time and share it with Crusoe support, without SSHing into the instance or assembling logs yourself. Collection runs on the instance through the Crusoe Watch Agent, and the resulting file is stored so you can download it or attach it to a support ticket.

Diagnostics are available for standalone VMs and for Crusoe Managed Kubernetes (CMK) nodes, and you can generate them from the console, the CLI, or the API.

Diagnostics currently cover these GPU bug reports:

  • NVIDIA bug reports (nvidia_bug_report): Include nvidia-smi output and kernel XID logs. Available for instances with NVIDIA GPUs.
  • AMD bug reports: Available for instances with AMD GPUs.

The report type appears in the Type field when you check a diagnostic's status.

Requirements​

Diagnostics require the Crusoe Watch Agent, which is installed by default at VM creation. Minimum versions depend on the GPU vendor and the instance type:

Report typeStandalone VMsCMK nodes
NVIDIAAgent version 1.0.3+Helm chart version 0.3.12+
AMDAgent version 1.0.5+Helm chart version 0.3.19+

For the full capability matrix and installation steps, see Get started.

Generate a Report from the Console​

  1. Navigate to Compute in the left navigation bar.
  2. Select your VM from the list.
  3. Click the three vertical dots icon next to the Start/Stop VM button, then click Generate bug report.

When collection completes, download the report or attach it to a support ticket.

Generate a report with the CLI​

The CLI calls the same API the Console uses, so reports generated either way appear in both places. This is useful for collecting reports across many instances at once, or for capturing diagnostics from a script when a job fails.

  1. Trigger a collection. The command returns a diagnostic ID:

    crusoe diagnostics vm create np-9addff51-1
    successfully created Diagnostic
    Diagnostic ID: 4d96bc56-d3ea-4314-8aee-35373bbd9798

  2. Check the status. Collection runs asynchronously on the instance, so poll until the status is completed:

    crusoe diagnostics vm status np-9addff51-1 \
      --diagnostic-id 4d96bc56-d3ea-4314-8aee-35373bbd9798
    Type: nvidia_bug_report
    Status: completed
    Created At: 2026-09-01T20:50:30Z
    Updated At: 2026-09-01T20:50:58Z

  3. Download the report. By default the file is written to the current directory as diagnostic-<diagnostic-id>.log.gz; pass --output to choose a different path:

    crusoe diagnostics vm download np-9addff51-1 \
      --diagnostic-id 4d96bc56-d3ea-4314-8aee-35373bbd9798
    downloaded diagnostic report to diagnostic-4d96bc56-d3ea-4314-8aee-35373bbd9798.log.gz

Both status and download need the VM name or ID and the diagnostic ID. If you don't have the diagnostic ID—for example, when the report was generated from the Console—look up the most recent completed report for the instance:

crusoe diagnostics vm latest np-9addff51-1
Diagnostic ID: 4d96bc56-d3ea-4314-8aee-35373bbd9798
Created At: 2026-09-01T20:50:30Z
Updated At: 2026-09-01T20:50:58Z

For the complete command syntax and flags, see the crusoe diagnostics CLI reference.

Collect Reports Across Multiple Instances​

To gather diagnostics from every VM in a project, trigger a collection on each one, then download the results:

for vm in $(crusoe compute vms list --json | jq -r '.[].name'); do
  crusoe diagnostics vm create "$vm"
done

Check the status before downloading, since collection takes several minutes per instance.

Auditing​

Diagnostic collection and download are control plane actions, so both are recorded in Audit Logs—whether performed from the console, the CLI, or the API. Each entry captures who performed the action, the instance it targeted, and whether it succeeded, giving you a 90-day history of diagnostic activity in your organization.

This means you can see who downloaded a diagnostic from a given instance. Audit logs are available to users with the admin role.

Report an issue​

To open a pre-filled support ticket that includes instance information and attaches the latest available bug report, use Report an issue from the same action menu in the console.

For other support channels, see Contact Support.

Collection error messages​

If collection fails, the following error messages will appear in the console or in the status output:

Error MessageCondition
Bug report script unavailableScript not found on the node
Bug report script execution timed outScript subprocess timed out
Bug report script failed with return code: {code}Script exited with non-zero return code
NVIDIA driver pod not foundNVIDIA driver pod not found on node
Error executing bug report scriptKubernetes API error during script execution
Bug report script returned no outputScript produced no expected output
Unexpected error downloading bug reportFailed to download log file from driver pod
Bug report generation timed outOverall collection timed out
Bug report upload failedUpload succeeded but result reporting failed
Internal Server ErrorUnknown error during collection

CLI errors​

The CLI also returns errors when the request itself can't be resolved:

MessageCondition
could not find VM with nameNo VM in the current project matches the name or ID you passed
could not get Diagnostic: client is not authorized to call this resource: unauthorized to access this VMYour credentials don't grant access to that VM, or it's in another project
No completed diagnostic found for this VM.The VM has no finished diagnostic yet
required flag(s) "diagnostic-id" not setstatus and download need --diagnostic-id in addition to the VM

What's next​