## Compiling a LLM for High Performance Inference This is the next blog in the **blog series** on LLMs and Inference. In the previous blog on LLMs and Inference, we discussed about the [safetensors](https://docs.rafay.co/blog/2025/04/23/safetensors-the-secure-scalable-format-powering-llm-inference/) format for LLMs. In this blog, we will walk through a critical step for LLM Inference. Compiling a Large Language Model (LLM) generally refers to optimizing the model’s **computational graph** and **kernel execution** to improve inference or training performance on specific hardware (like GPUs or TPUs). Think of this as the next logical step that is performed after loading a model. ## Safetensors: The Secure, Scalable Format Powering LLM Inference As Large Language Models (LLMs) like LLaMA, Mistral, and DeepSeek continue to scale into the hundreds of billions of parameters, model efficiency becomes as important as model quality. One often-overlooked bottleneck is the **model loading format**. This is one of the primary focus areas for [**safetensors**](https://huggingface.co/docs/safetensors/en/index). ## End-User Self-Service for Automated User Profile Creation in SageMaker Domains As organizations expand their use of Amazon SageMaker to empower data scientists and machine learning (ML) engineers, managing access to **development environments** becomes a critical concern. In the [last blog](https://docs.rafay.co/blog/2025/04/17/why-enterprises-should-use-domains-for-sagemaker-ai/), we discussed how SageMaker Domains can provide isolated, secure, and fully-featured environments for users. However, **manually** creating user profiles for every user quickly becomes a bottleneck—especially in large or fast-growing organizations. Asking users to submit an IT ticket and wait for days before it can be fulfilled is unacceptable in today's fast-paced environment. In this blog, we will describe how organizations use Rafay's GPU PaaS to provide their users with a **self-service experience** to onboard themselves into SageMaker Domains without waiting on IT or platform teams. This not only improves efficiency and user experience but also ensures consistency and compliance across the organization. ## Why Enterprises Should Use Domains for SageMaker AI As organizations continue to invest in artificial intelligence (AI) and machine learning (ML) to drive digital transformation, the demand for streamlined, secure, and scalable development environments has never been greater. Many organizations that are standardized on Amazon AWS may use [**Amazon SageMaker AI**](https://aws.amazon.com/sagemaker-ai/) to build, train, and deploy machine learning models at scale with minimal operational overhead. SageMaker AI provides a **fully managed** environment that streamlines the entire ML lifecycle, enabling faster innovation, stronger governance, and cost-effective AI development. In this introductory blog, we will describe one of the most critical capabilities of SageMaker AI called **Domains**. In the [next blog](https://docs.rafay.co/blog/2025/04/18/end-user-self-service-for-automated-user-profile-creation-in-sagemaker-domains/), we will describe how organizations can scale their AI/ML teams by providing their data scientists and ML engineers with a self-service experience for access to SageMaker Domains. ## Introduction to Slurm-The Backbone of HPC This is part-1 in a blog series on [**Slurm**](https://slurm.schedmd.com/documentation.html). In the first part, we will provide some introductory concepts about Slurm. We are not talking about the fictional soft drink in the world of [Futurama](https://en.wikipedia.org/wiki/Futurama). Instead, this blog is about Slurm (Simple Linux Utility for Resource Management), an open-source, fault-tolerant, and highly scalable **cluster management** job scheduler and resource manager used in high-performance computing (HPC) environments. Slurm was originally conceptualized in 2002 at [Lawrence Livermore National Laboratory](https://www.llnl.gov/) (LLNL) and has been actively developed and maintained especially by [SchedMD](https://www.schedmd.com/). In this time, Slurm has become the defacto workload manager for HPC with >50% of the Top-500 supercomputers using it. ## Fractional GPUs using Nvidia's KAI Scheduler At KubeCon Europe, in April 2025, Nvidia announced and launched the **Kubernetes AI (KAI) [Scheduler](https://github.com/NVIDIA/KAI-Scheduler)**. This is an Open Source project maintained by [Nvidia](https://www.nvidia.com/). The KAI Scheduler is an advanced Kubernetes scheduler that allows administrators of Kubernetes clusters to **dynamically allocate** GPU resources to workloads. Users of the Rafay Platform can immediately leverage the KAI scheduler via the integrated [**Catalog**](https://docs.rafay.co/catalog/catalog_overview/). To help you understand the basics quickly, we have also created a brief video introducing the concepts and a live demonstration showcasing how you can allocate **fractional** GPU resources to workloads. Fractional GPU Sharing with NVIDIA KAI Scheduler on Kubernetes | Rafay - YouTube ## How to Select the Right GPU for Open Source LLMs? Deploying and operating an **open-source Large Language Model (LLM)** requires careful planning when selecting the right **GPU model and memory capacity**. Choosing the optimal configuration is crucial for performance, cost efficiency, and scalability. However, this process comes with several challenges. In this blog, we will describe the factors that you need to consider to select the optimal GPU model for your LLM. We have also published a table capturing optimal GPU models to deploy and use Top-10 open source LLMs. ## Developer Self Service Access to DeepSeek on Amazon EKS A few weeks back, [**Tiago Reichert**](https://community.aws/@tiagoreichert) from [AWS](https://aws.amazon.com/) published a very interesting [blog](https://community.aws/content/2sJofoAecl6jVdDwVqglbZwKz2E/hosting-deepseek-r1-on-amazon-eks) on AWS Community showcasing how you can deploy and use the [**DeepSeek-R1**](https://api-docs.deepseek.com/news/news250120) LLM on an Amazon EKS Cluster operating in Auto Mode. Detailed step-by-step instructions for this are documented in this [Git Repo](https://github.com/aws-samples/deepseek-using-vllm-on-eks). In this blog, we will describe how we took AWS's excellent blog and packaged it to provide a **turnkey, 1-click self-service experience** for non-AWS administrator type users in a typical enterprise. It took one of our solution architects 30 minutes to wrap AWS's example code using Rafay's **Environment Manager and PaaS**. ## Using Hubble and Cilium with Rafay MKS based Kubernetes Cluster for Data Centers In our **[first blog](https://docs.rafay.co/blog/2025/03/03/supercharge-kubernetes-networking-observability-using-hubble-and-cilium/)** about Hubble for Cilium, we reviewed a real-life example highlighting where traditional monitoring tools fall short. We then looked at how Hubble + Cilium can address these gaps. In the **[second blog](https://docs.rafay.co/blog/2025/03/04/turnkey-integration-with-cilium-cni/)**, we discussed how Rafay provides our customers with a tight, turnkey integration with Cilium for various cluster types (i.e., Rafay MKS for Data Centers and Public Cloud Distributions such as Amazon EKS). In this [get started guide](https://docs.rafay.co/learn/quickstart/mks/mks_hub_cluster/overview/), we will review how a platform engineer can configure, deploy, and use Hubble for Cilium on a Rafay MKS Kubernetes cluster operating in a data center (aka on-premises environment). The three high-level steps are: 1. Provision an Upstream Kubernetes Cluster in your data center using Rafay MKS 2. Configure and Deploy Cilium CNI as a software add-on in a Cluster Blueprint (i.e., Bring Your Own CNI) 3. Use Hubble to observe network flows ## Turnkey Integration with Cilium CNI In the [**first blog**](https://docs.rafay.co/blog/2025/03/03/supercharge-kubernetes-networking-observability-using-hubble-and-cilium/), we discussed how organizations can use Hubble for Cilium for observability. In this **blog**, we will look at how the Rafay Platform provides a tight, turnkey integration with **Cilium** making life easy for platform teams. In the next [blog](https://docs.rafay.co/blog/2025/03/06/using-hubble-and-cilium-with-rafay-mks-based-kubernetes-cluster-for-data-centers/), my colleague will describe and showcase how an administrator can configure and enable Hubble on a Rafay MKS based Kubernetes cluster with the Cilium CNI.