Serverless Inference for Production AI

Run models on-demand without managing infrastructure

Serverless inference gives developers instant, on-demand access to AI models without provisioning or managing GPUs, clusters, or runtime environments. Instead of waiting on infrastructure, teams can deploy, scale, and operate inference workloads through simple APIs, accelerating the path from model to production.

With the Rafay Platform, serverless inference delivers a consistent, governed, and production-ready experience for AI workloads across teams, tenants, and environments.

How Cloud Providers can provide Multi-Tenant, Serverless Inference to their Customers

Multi-Tenant Serverless Inference for Cloud Providers with the Rafay Platform's GPU PaaS - YouTube

Serverless Inference FAQs

What counts as a node?

A node is a physical or virtual server/machine.

Do you have any volume discounts?

Yes! As the number of nodes increases the price per cluster or per node decreases.

What about short-lived or ephemeral clusters?

Our customers love to experiment, and we don’t ding them for it. We don’t charge for node count spikes, but look at the running average of nodes in use when calculating usage.

Is there a difference between production and non-production pricing?

The management overhead for helping our customers operate dev vs prod clusters is effectively the same, so we treat all nodes the same.

What if I use more nodes than I’ve licensed?

Rafay has a true-up forward policy, meaning that we don’t carry out chargebacks for scenarios where the consumption in a completed billing cycle exceeded the licensed count. If the new, steady-state number of nodes is expected to be higher, our customer success team will discuss the situation with you, and take steps to adjust billing accordingly for the next billing cycle.

How much does Enterprise Support (24x7x365) cost?

Enterprise Support is available at an additional fee equaling 20% of the cluster or node subscription.

Do you have EDU or GOV discounts?

Yes, please contact sales for more information about discounts for educational institutions and government agencies.

What is serverless inferencing?

Serverless inference allows teams to deploy and run AI models without provisioning or managing underlying infrastructure. Instead of configuring clusters or managing GPUs, developers interact with simple APIs that scale automatically based on demand.

Rafay turns GPU infrastructure into on-demand inference services—eliminating operational friction and accelerating time to production.

What is an AI token factory?

An AI Token Factory is the operating layer that transforms GPU infrastructure into governed, consumable AI services.

Instead of exposing raw GPUs or unmanaged clusters, organizations deliver production-ready model APIs that are:

• Token-metered for transparent usage tracking
• Multi-tenant with strict isolation and RBAC
• Quota-controlled to prevent runaway spend
• Governed by policy and compliance guardrails
• Monetizable through usage-based billing

Serverless inference is how models are delivered. A Token Factory is how they are scaled, controlled, and turned into repeatable services.

What is an AI inference platform?

An AI inference platform is a scalable environment for deploying and managing AI models in production. It handles request routing, GPU allocation, scaling, monitoring, and performance optimization. In enterprise environments, inference platforms are critical for supporting token factories that must generate tokens reliably and efficiently at scale.

How does LLM token generation work?

LLM token generation works by tokenizing an input prompt, running it through a trained neural network, and predicting the next most probable token. This process repeats sequentially until the full response is produced. Each new token is influenced by the tokens that came before it, which allows models to generate coherent text.

What is an inference engine in AI?

An inference engine is the system that runs a trained AI model to generate predictions or text in real time. In large language models, the inference engine processes input tokens and produces output tokens. Its efficiency directly impacts response speed, scalability, and cost per token.

What role does Rafay play in AI factories?

Rafay provides the control plane for AI factories, handling orchestration, multi-tenancy, governance, and self-service access to AI infrastructure across cloud, on-prem, and sovereign environments.

Is Rafay an AI factory?

Rafay is not a GPU manufacturer or model provider. Rafay provides an infrastructure orchestration and consumption platform that enables organizations to operate AI factories by turning AI infrastructure into a governed, self-service platform. Learn more about AI factories here: https://rafay.co/ai-and-cloud-native-blog/what-is-an-ai-factory

Serverless Inference, Built for Production AI

Rafay enables GPU clouds and enterprises to deliver model inference as an on-demand service without exposing infrastructure complexity.

Plug-and-Play LLM Integration

Instantly deliver popular open-source LLMs (e.g., Llama 3.2, Qwen, DeepSeek) using OpenAI-compatible APIs to your customer base—no code changes required.

Serverless Access

Deliver a hassle-free, serverless experience to your customers looking for the latest and greatest GenAI models.

Token-Based Pricing & Visibility

Flexible usage-based billing with complete cost transparency and historical usage insights.

Secure & Auditable API Endpoints

HTTPS-only endpoints with bearer token authentication, full IP-level audit logs, and token lifecycle controls.

Why DIY when you can FLY with the Rafay Platform serverless inference offering?

Most organizations have invested in GPU infrastructure but struggle to make it usable for real-world AI applications. Rafay transforms raw compute into fully operational inference services by enabling instant model deployment as API endpoints, eliminating manual provisioning, automatically scaling based on demand, and optimizing GPU utilization across shared environments. By abstracting infrastructure complexity, teams can focus on building and deploying AI applications instead of managing systems.

Pre-optimized interference templates

Intelligent auto-scaling of GPU resources

Enterprise-grade security and token authentication

Built-in observability, cost tracking, audit logs

How Serverless Inference Connects to Token Factory

Serverless inference powers the execution of AI workloads. Token Factory builds on top of it to enable consumption and monetization.

• Serverless inference → runs models as APIs
• Token Factory → tracks and meters usage via tokens

Together, they enable organizations to move from running models to delivering AI as a service.

Start with serverless inference to operationalize models. Extend to Token Factory to:

• Deliver OpenAI-style APIs
• Track usage across teams or customers
• Monetize AI services with consumption-based pricing

Featured Resources

AI Token Factory

AI Token Factory extends the Rafay Platform to deliver AI services through APIs and token-metered consumption. Production-ready AI APIs run on GPU infrastructure while maintaining governance, multi-tenancy, and operational control. Token-metered consumption provides visibility into usage and enables internal chargeback or monetization models.

How Telecom Provider Telus Built an AI Factory

One of Canada’s Largest Telecom Companies, TELUS, Launches a Sovereign, Developer-Ready AI Studio Powered by Rafay

Building AI Value within Borders

Rafay's central orchestration platform facilitates efficient, self-service infrastructure and AI application management.

The CIO’s guide to scalable, compliant, and developer-ready AI deployment

Orchestrating the future of AI: The CIO’s guide to scalable, compliant, and developer-ready AI deployment

Unlock Your AI Potential with Cisco and Rafay: Transform AI PODs into a Self-Service GPU Cloud

Cisco provides AI-optimized infrastructure. Rafay makes it usable across teams, tenants, and use cases in days.

The Definitive GPU PaaS Reference Architecture

Understand what it takes to deliver the right GPU infrastructure to your business.

Operationalizing AI Fabrics with Aviz ONES, NVIDIA Spectrum-X, and Rafay

Discover the new AI operations model available to enterprises that enables self-service consumption and cloud-native orchestration for developers.

GPU cloud evaluation report

Evaluating how the Rafay Platform delivers a GPU cloud for enterprises and cloud service providers by PivotNine.