Get Started with Token Factory - Rafay Product Documentation

Basics

In this guide you will deploy an instance of an open source model using Rafay's Token Factory and make it available to a customer/tenant for use as an inference endpoint.

The Token Factory provides operators with an integrated environment to onboard, manage, and deploy LLMs on their GPU-backed infrastructure. The service includes resources that support the complete model lifecycle, from preparing compute capacity to publishing models for organizational use.

Assumptions

This exercise assumes the following requirements are in place.

Info

To ensure this guide does not impose requirements for expensive GPU hardware, we have optimized this for Qwen2.5-0.5B, a very small, lightweight model that requires minimal GPU memory (2-3GB VRAM). This makes it ideal for low-end GPUs, consumer hardware, or Edge devices.

1. Create Compute Cluster

In this section, you will import/register a Kubernetes cluster with GPUs into the Token Factory.

kubectl apply -f <compute-name>-compute-bootstrap.yaml

Once all resources in the namespaces gaap-controller and monitoring are in a running state, the cluster will have a status of "Success"

2. Create Endpoint

In this section, you will create an endpoint which represents the access point through which inference requests are served by the Token Factory. Endpoints route incoming traffic to deployed models running on a GPU-enabled compute cluster.

Info

In this guide, we will select "Internal" since we will be using Cluster IP. If you have a public IP and a load balancer, select "External".

Once the endpoint is created, the IP address associated with the endpoint will be visible.

Important

Ensure this IP address is associated in DNS with the hostname used. This will allow proper domain name resolution when users are accessing the endpoint.

3. Create Provider

In this section, you will create a provider. A Provider represents the source or organization from which the LLM originates. Examples include Llama (Meta), Qwen, NVIDIA, Google, and custom enterprise providers.

4. Create Model

In this section, you will create a model. A model represents the LLM that the administrator onboard into the system. Each model includes core metadata such as name, description, provider, and use case. This information determines how the model is organized, how it appears in the console, and how it will later be deployed for inference.

LLM Use Case

Info

For the model to support chat protocol, vLLM requires the model to include a chat template in its tokenizer configuration. The chat template is a Jinja2 template that specifies how roles, messages, and other chat-specific tokens are encoded in the input.

Model Selection and Location

In this guide, the deployments will download the model weights during deployment from HuggingFace. For production deployments, operators will prefer to use a local storage namespace to ensure deployments are blazing fast.

Important

A newly created model has zero deployments by default. One or more deployments can be created for the same model, each with different endpoints or runtime configurations.

5. Model Sharing

Next, you will share the model with a downstream Tenant Organization to be used by end users.

6. Create Model Deployment

In this section, you will create a model deployment. Model Deployments define how a LLM is deployed for inference. A deployment binds a model to an endpoint, selects an inference engine, and configures runtime resources such as replicas, CPU, memory, and GPU.

Engine Selection

Note

Operators can select the version of vLLM they wish to use. The vLLM image can be pretty large in size (Gigabytes) and operators may wish to host the image in a local container registry to ensure fast/reliable deployments.

Resources

Specify the cpu, memory, gpu and storage resources that you wish to allocate to vLLM

Scale

In this step, the operator makes a determination of scale for the deployment. With multiple replicas, traffic can be load balanced across the replicas delivering higher scale.

Metering

In this section, the operator will specify how usage will be metered. They will specify the currency and the rate for every million tokens. The Rafay Token Factory counts input and output tokens separately.

After a few minutes, the model will be deployed to the specified cluster.

7. Model Deployment Sharing

Next, you will share the model with a downstream Tenant Organization to be used by end users.

8. End User Utilization

Finally, you will use a tenant end user account and utilize the inference endpoint. In this guide, we will execute the commands directly from the Kubernetes cluster.

export API_KEY=<API KEY VALUE>

{
  "id": "chatcmpl-c99463c4-28bd-46b3-a2de-20c410e5a506",
  "object": "chat.completion",
  "created": 1771890924,
  "model": "gs-deployment",
  "choices": [\
    {\
      "index": 0,\
      "message": {\
        "role": "assistant",\
        "content": "There are several open-source inference libraries available, but one of the most widely used and well-regarded is PyTorch. It has been developed by Facebook and is known for its simplicity and ease of use.\nPyTorch includes a large number of pre-trained models that can be used as starting points for building more complex models. Some popular PyTorch models include ResNet, VGG, and SSD (Segmentation Dynamics).\nAnother open-source inference library is Keras, which is an extension of TensorFlow. While it does not have the same level of support as PyTorch, Keras offers many similar features and is easy to learn.\nBoth PyTorch and Keras are excellent choices for developing and training machine learning models on a variety of hardware platforms.",\
        "refusal": null,\
        "annotations": null,\
        "audio": null,\
        "function_call": null,\
        "tool_calls": [],\
        "reasoning": null,\
        "reasoning_content": null\
      },\
      "logprobs": null,\
      "finish_reason": "stop",\
      "stop_reason": null,\
      "token_ids": null\
    }\
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 30,
    "total_tokens": 184,
    "completion_tokens": 154,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

After a few minutes, the usage metrics will be populated and the user can view their historical usage and total spend.