Understanding LLM Inference Metrics in Rafay's Token Factory

Understanding Model Deployment Metrics in Rafay's Token Factory

March 27, 2026

When you're running LLM inference at scale, "the model works" is table stakes. What separates a demo from a production service is knowing how well your models perform under real-world conditions — how fast users see the first token, whether streaming feels natural, and whether your infrastructure is meeting the service-level objectives you've committed to. That's exactly where inference metrics come in.

Rafay's Token Factory transforms raw GPU infrastructure into governed, consumable AI services. It enables organizations to deploy models from sources like Hugging Face or NVIDIA NGC as production-grade APIs in minutes, with built-in multi-tenancy, token-metered billing, and auto-scaling. But shipping a model as an API is only half the story.

The other half is observability: knowing, in real time, whether your inference endpoints are performing within acceptable bounds. The Token Factory's built-in metrics dashboard gives operators exactly this visibility — surfacing the key latency, throughput, and resource utilization metrics that matter most.

The Metrics That Matter for LLM Inference

Before diving into the Rafay dashboard, it helps to understand the core metrics categories for any LLM inference system. These fall into four groups: latency metrics, throughput metrics, percentile metrics, and resource utilization metrics. Each answers a different question about your system's health.

1. Latency Metrics: What Users Actually Feel

Latency is the metric class that directly impacts user experience. There are three complementary latency metrics, each answering a different question about the request lifecycle.

TTFT — Time to First Token

TTFT measures the elapsed time between when a request is submitted and when the very first token of the response arrives. It captures three things: queue wait time, the model's prefill computation, and network overhead.

Why it matters: TTFT is what users feel first. In a chatbot, coding assistant, or any interactive application, a long TTFT creates a perception of lag before anything starts appearing on screen. For interactive workloads, the general industry target is a p95 TTFT under 500ms. Anything above that, and users start wondering if the system is broken.

What drives it up: longer input prompts, high queue depth under load, or insufficient GPU capacity for the model size.

ITL — Inter-Token Latency (also called TBT or TPOT)

ITL measures the time between consecutive generated tokens during the decode phase. It tells how smooth the response feels as it streams. Human reading speed is roughly 4–5 tokens per second, which means an ITL up to about 200ms is acceptable. Above 250ms, streaming starts to feel choppy or broken.

2. How Rafay Surfaces These Metrics

In the Rafay Token Factory, each model deployment gets its own dedicated Metrics tab. The dashboard is designed to give operators both an at-a-glance summary and deep time-series visibility.

The Summary Cards

The Time-Series Charts

3. Percentile Metrics: Why Averages Will Mislead You

One of the most important things the Rafay dashboard does is display metrics at multiple percentile levels rather than just averages.

p50 (Median)

The median represents the typical user experience — 50% of requests are faster, 50% are slower. But it's terrible for SLAs.

p95

Captures what your "unlucky" 5% of users experience — and in production, that 5% adds up to a lot of real people.

p99

Reveals near-worst-case performance. If your p99 is healthy and consistent, you can be confident your system is stable.

4. KV Cache: The Metric Most Teams Miss

The KV cache metric is arguably the most important resource-level indicator for LLM inference. The KV cache grows with both input length and output length.

5. Optimizing by Workload Type

Not all inference workloads are created equal. The metrics you prioritize should depend on what you're building.

Interactive Workloads

For interactive applications, monitor TTFT p95 < 500ms and ITL p95 < 250ms.

Batch and Offline Workloads

For batch processing, the north star metrics are Tokens Per Second (TPS) and cost per million tokens.

6. Reading the Dashboard: A Practical Walkthrough

At a glance: The summary cards show health metrics and alerts.

7. From Metrics to Action

Metrics are only valuable if they drive decisions. Here's a quick reference for common issues with solutions:

TTFT is high: Check queue depth and request arrival rate.

ITL is degrading: Look at KV cache utilization and GPU memory bandwidth.

E2E latency exceeds SLO: Decompose into TTFT + (ITL × tokens).

KV Cache near capacity: Consider adding replicas or reducing max sequence length.

Conclusion

Running LLM inference in production isn't just about deploying a model — it's about continuously understanding and optimizing how that model performs. The key takeaways are: