Understanding LLM Inference Metrics in Rafay's Token Factory

Understanding Model Deployment Metrics in Rafay's Token Factory

March 27, 2026

When you're running LLM inference at scale, "the model works" is table stakes. What separates a demo from a production service is knowing how well your models perform under real-world conditions — how fast users see the first token, whether streaming feels natural, and whether your infrastructure is meeting the service-level objectives you've committed to. That's exactly where inference metrics come in.

Rafay's Token Factory transforms raw GPU infrastructure into governed, consumable AI services. It enables organizations to deploy models from sources like Hugging Face or NVIDIA NGC as production-grade APIs in minutes, with built-in multi-tenancy, token-metered billing, and auto-scaling. But shipping a model as an API is only half the story.

The other half is observability: knowing, in real time, whether your inference endpoints are performing within acceptable bounds. The Token Factory's built-in metrics dashboard gives operators exactly this visibility — surfacing the key latency, throughput, and resource utilization metrics that matter most.

The Metrics That Matter for LLM Inference

Before diving into the Rafay dashboard, it helps to understand the core metrics categories for any LLM inference system. These fall into four groups: latency metrics, throughput metrics, percentile metrics, and resource utilization metrics. Each answers a different question about your system's health.

1. Latency Metrics: What Users Actually Feel

Latency is the metric class that directly impacts user experience. There are three complementary latency metrics, each answering a different question about the request lifecycle.

TTFT — Time to First Token

TTFT measures the elapsed time between when a request is submitted and when the very first token of the response arrives. It captures queue wait time, the model's prefill computation, and network overhead.

Why it matters: TTFT is what users feel first. In a chatbot or any interactive application, a long TTFT creates a perception of lag before anything starts appearing on screen.

ITL — Inter-Token Latency (also called TBT or TPOT)

ITL measures the time between consecutive generated tokens during the decode phase. While TTFT tells you how long before the response starts, ITL tells you how smooth the response feels as it streams.

2. How Rafay Surfaces These Metrics

In the Rafay Token Factory, each model deployment gets its own dedicated Metrics tab within the deployment detail view. The dashboard is designed to give operators both an at-a-glance summary and deep time-series visibility.

The Summary Cards

At the top of the metrics dashboard, four summary cards provide a quick health check:

The Time-Series Charts

Below the summary cards, the dashboard presents four detailed time-series charts, each plotting values across p50, p90, p95, and p99 percentiles over time:

3. Percentile Metrics: Why Averages Will Mislead You

One of the most important things the Rafay dashboard does is display metrics at multiple percentile levels rather than just averages.

p50 (Median)

The median represents the typical user experience — 50% of requests are faster, 50% are slower.

p95

Captures what your "unlucky" 5% of users experience.

p99

Reveals near-worst-case performance. If your p99 is healthy and consistent, you can be confident your system is stable.

4. KV Cache: The Metric Most Teams Miss

The KV cache metric is less well-known than latency metrics, but it's arguably the most important resource-level indicator for LLM inference. The KV cache stores the key-value pairs computed during the attention mechanism, enabling the model to remember context during token generation.

5. Optimizing by Workload Type

Metrics you prioritize should depend on what you're building.

Interactive Workloads (Chat, Agents, Coding Assistants)

The north star metric is TTFT p95 < 500ms, followed closely by ITL p95 < 250ms.

Batch and Offline Workloads (Pipelines, Evals, Data Generation)

For batch processing, latency is secondary to efficiency. The north star metrics are Tokens Per Second (TPS) and cost per million tokens.

6. Reading the Dashboard: A Practical Walkthrough

Let's walk through what a real Rafay Token Factory metrics dashboard tells us, using the example of a Qwen3 Coder model deployed with NVIDIA Dynamo as the inference engine.

7. From Metrics to Action

Metrics are only valuable if they drive decisions. Here's a quick reference for what to do when metrics go sideways:

TTFT is high

Check queue depth and request arrival rate. Consider adding replicas or enabling prefix caching.

ITL is degrading

Look at KV cache utilization and GPU memory bandwidth.

E2E latency exceeds SLO

Identify which component is contributing most and address accordingly.

KV Cache near capacity

Add replicas or reduce max sequence length.

Conclusion

Running LLM inference in production isn't just about deploying a model — it's about continuously understanding and optimizing how that model performs under real-world conditions. The key takeaways: