Metrics

Serverless Inference records metrics for every model it serves, with no configuration required. The metrics update every minute and are available on the metrics page. You can also integrate them with Grafana dashboards through a Prometheus-compatible query API. To query that API yourself, see Retrieve metrics using the PromQL API.

Available metrics

The following queries return the same series that back the charts on the console metrics page. Replace {project_id} with your project ID and {model_alias} with the model name you pass in your API requests.

Requests

inference_counter_chat_request counts only successfully served requests. Use inference_counter_all_chat_request to include error and rate-limited responses.

Metric	Definition	Metric query
Request rate	Successful requests per second	sum by (model_alias) ( rate( inference_counter_chat_request{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )
Requests by status code	All requests, including client errors, rate limits (429), and server errors, split by HTTP status	sum by (model_alias, status_code) ( rate( inference_counter_all_chat_request{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )
Total requests	Cumulative count of all requests over the query window	sum by (model_alias) ( increase( inference_counter_all_chat_request{project_id="{project_id}", model_alias="{model_alias}"}[24h] ) )

Tokens

These queries cover input and output token throughput, prefix cache effectiveness, and cumulative totals over a window.

Metric	Definition	Metric query
Input token rate	Input (prompt) tokens processed per second	sum by (model_alias) ( rate( inference_counter_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )
Output token rate	Output (completion) tokens generated per second	sum by (model_alias) ( rate( inference_counter_output_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )
Cached input tokens	Input tokens served from the prefix cache, per second	sum by (model_alias) ( rate( inference_counter_cached_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) )
Prefix cache hit rate	Cached input tokens as a percentage of all input tokens	( sum by (model_alias) ( rate( inference_counter_cached_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) / sum by (model_alias) ( rate( inference_counter_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 100
Tokens per minute (TPM)	Total tokens (input plus output) processed per minute	( sum by (model_alias) ( rate( inference_counter_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) + sum by (model_alias) ( rate( inference_counter_output_token{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 60
Total input tokens	Cumulative input tokens over the query window	sum by (model_alias) ( increase( inference_counter_prompt_token{project_id="{project_id}", model_alias="{model_alias}"}[24h] ) )
Total output tokens	Cumulative output tokens over the query window	sum by (model_alias) ( increase( inference_counter_output_token{project_id="{project_id}", model_alias="{model_alias}"}[24h] ) )

Latency

Latency histograms record values in seconds. Multiply by 1000 for milliseconds, as the console does. Each query below returns the median—substitute 0.9, 0.95, or 0.99 for 0.5 to get other percentiles.

Every histogram on this page, including the per-request token histograms below, also exposes _sum and _count series, so an average over a window is sum(increase(<metric>_sum[1h])) / sum(increase(<metric>_count[1h])).

Metric	Definition	Metric query
Time to first token (TTFT)	Time from request receipt to the first returned token	histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_first_token_latency_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 1000
Time per output token (TPOT)	Time between successive output tokens	histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_output_token_latency_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 1000
End-to-end latency	Total request duration, from request receipt to completed response	histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_chat_latency_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) ) * 1000

Tokens per request

These histograms show how large individual requests are. As with the latency histograms, substitute another quantile for 0.5 to see a different percentile.

Metric	Definition	Metric query
Input tokens per request	Distribution of input (prompt) tokens per request	histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_input_tokens_per_request_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) )
Output tokens per request	Distribution of output (completion) tokens per request	histogram_quantile( 0.5, sum by (model_alias, le) ( rate( inference_histogram_output_tokens_per_request_bucket{project_id="{project_id}", model_alias="{model_alias}"}[5m] ) ) )

Labels

Use these labels to filter and group the queries above:

Label	Description
project_id	The project billed for the request
model_alias	The model name you pass in your API request—filter on this label
model_name	The deployment serving the request: the upstream model name plus a deployment suffix, such as google-gemma-4-31b-it-0dc46fd7. It doesn't match model_alias
service_tier	The service tier that handled the request
status_code	HTTP status code of the response. Present on both request counters, but inference_counter_chat_request only ever reports 200, so query inference_counter_all_chat_request for errors
is_streaming	Whether the request used streaming

Retrieve metrics using the PromQL API

Query the metrics API endpoint directly to retrieve data for a single instant or a specific time range. The API endpoint is:

https://api.cloud.crusoe.ai/v1/projects/<project-id>/metrics/timeseries

To find your project ID, navigate to the projects page in the console, and copy the project ID.

Generate a monitoring token

Querying metrics requires a monitoring token. To generate one, run the following command with the Crusoe CLI:

crusoe monitoring tokens create

This command generates an API-Key that authenticates your requests to the metrics API. Store the token in a secret or key management tool, because you can't retrieve it later.

Query metrics

To retrieve the most recent TTFT data point in your project, run:

curl -G https://api.cloud.crusoe.ai/v1/projects/<project-id>/metrics/timeseries \
  --data-urlencode 'query=histogram_quantile(0.5, sum by (model_alias, le) (rate(inference_histogram_first_token_latency_bucket[5m]))) * 1000' \
  -H 'Authorization: Bearer <API-Key>'

Import data into Grafana

To chart Serverless Inference metrics alongside the rest of your telemetry, add a Prometheus data source to your own Grafana instance. You need the API-Key from Generate a monitoring token.

1. Set Prometheus server URL to https://api.cloud.crusoe.ai/v1/projects/<project-id>/metrics/timeseries.

2. Under Authentication → HTTP headers, add the following header:

   Header: Authorization
   Value: Bearer <API-Key>