Prerequisites

This document introduces the prerequisites for the MoAI Inference Framework and provides instructions on how to install them.

info

To follow this document, you need to understand the configuration of the Kubernetes cluster where the MoAI Inference Framework will be installed. Since Moreh provides support for installing the MoAI Inference Framework at customer sites, if you encounter any difficulties, you can request assistance from the Moreh team.

---

Target system

To install the MoAI Inference Framework, you must have

• Kubernetes 1.29 or later
• At least one worker node equipped with accelerators supported by the MoAI Inference Framework (e.g., AMD GPUs)
• cluster-admin privilege for the Kubernetes cluster
• A StorageClass defined in the Kubernetes cluster (required for storing the monitoring metrics, model weights, etc.)
• A Docker private registry accessible from the Kubernetes cluster

---

cert-manager

cert-manager is a powerful and extensible X.509 certificate controller for Kubernetes workloads. It is essential for managing TLS certificates within the MoAI Inference Framework. Both the Odin and Heimdall operators rely on cert-manager for their admission webhooks, so it must be installed first.

Deploy cert-manager using the following command:

helm upgrade -i cert-manager oci://quay.io/jetstack/charts/cert-manager \
    --version v1.18.4 \
    -n cert-manager \
    --create-namespace \
    --set crds.enabled=true

---

moai-inference-framework

The moai-inference-framework Helm chart deploys the infrastructure and observability dependencies required by the MoAI Inference Framework (such as monitoring, logging, and object storage), excluding GPU and network-related components. To deploy it, you first need to add Moreh's Helm chart repository.

helm repo add moreh https://moreh-dev.github.io/helm-charts

If you have already added the repository, make sure to update it.

helm repo update moreh

The container images for the MoAI Inference Framework are distributed through a private repository on Amazon ECR 255250787067.dkr.ecr.ap-northeast-2.amazonaws.com, and you need to obtain an authorization token to download them. To facilitate this, the moai-inference-framework chart defines the installation of an ECR token refresher. You need to specify your AWS credentials in the values file to configure the ECR token refresher.

info

If you pull images through a local registry such as Harbor instead of directly from ECR, disable the ECR token refresher by setting ecrTokenRefresher.enabled to false. See Container image caching with Harbor for details.

warning

The AWS credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) should have been provided to you along with your purchase or trial issuance of the MoAI Inference Framework. If you did not receive this information, please contact your point of purchase separately.

Create a moai-inference-framework-values.yaml file as follows. You need to replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your own values.

moai-inference-framework-values.yaml

ecrTokenRefresher:
  aws:
    accessKeyId: <AWS_ACCESS_KEY_ID>
    secretAccessKey: <AWS_SECRET_ACCESS_KEY>

info

If an infrastructure dependency such as kube-prometheus-stack is already installed in your cluster, skip its installation by setting the corresponding value to false. Refer to moai-inference-framework README for the full list of dependencies.

Then, deploy the moai-inference-framework chart using the following command:

helm upgrade -i mif moreh/moai-inference-framework \
    --version v0.5.0 \
    -n mif \
    --create-namespace \
    -f moai-inference-framework-values.yaml

---

Chart versions

The Heimdall and Odin charts installed below are pinned to the following released versions from the moreh Helm repository. The helm upgrade commands in the sections that follow use these same versions. (moai-inference-framework and moai-inference-preset are pinned directly in their own install commands.)

Heimdall

Chart	Version	App version
heimdall	v0.5.0	v0.5.0
heimdall-aigateway-crd	v0.10.0	v0.10.0
heimdall-crd	v0.5.0	v0.5.0

Odin

Chart	Version	App version
odin	v0.12.0	v0.12.0
odin-crd	v0.12.0	v0.12.0

---

Custom resource definitions

Install the custom resource definitions (CRDs) used by the operators. CRDs are packaged as separate charts so their lifecycle is independent of the operators.

helm upgrade -i odin-crd moreh/odin-crd --version v0.10.0 -n mif
helm upgrade -i heimdall-crd moreh/heimdall-crd --version v0.4.0 -n mif
helm upgrade -i heimdall-aigateway-crd moreh/heimdall-aigateway-crd --version v0.6.0 -n mif

This installs the custom resource definitions used by the operators, including InferenceService and InferenceServiceTemplate (Odin), AIGateway (Heimdall), and SchedulingProfile (Heimdall AIGateway).

---

Odin

Odin is the operator that manages inference services — the vLLM pods that serve a model. Install it after its CRDs and cert-manager.

helm upgrade -i odin moreh/odin \
    --version v0.10.0 \
    -n mif \
    --create-namespace

info

Odin installs LeaderWorkerSet (LWS) as a dependency. If LWS is already installed in your cluster, set --set lws.enabled=false.

---

Heimdall

Heimdall is the operator that runs the AI Gateway. It reconciles AIGateway resources into a gateway deployment and injects the gateway sidecar into inference pods. Because it injects sidecars into Odin-managed pods, install it after Odin's InferenceService CRD.

helm upgrade -i heimdall moreh/heimdall \
    --version v0.8.0 \
    -n mif \
    --create-namespace

---

moai-inference-preset

The moai-inference-preset Helm chart deploys the presets for the MoAI Inference Framework. The presets define preconfigured ways to run inference containers (e.g., Moreh vLLM containers) in the MoAI Inference Framework.

helm upgrade -i moai-inference-preset moreh/moai-inference-preset \
    --version v0.5.0 \
    -n mif

---

AMD GPU operator

This section describes how to set up the AMD GPU Operator on a Kubernetes cluster. See AMD GPU Operator / Kubernetes (Helm) for more details.

Add the ROCm's GPU Operator Helm chart repository.

helm repo add rocm https://rocm.github.io/gpu-operator
helm repo update rocm

Create a namespace for the AMD GPU Operator.

kubectl create namespace amd-gpu

During the installation of the AMD GPU Operator, the GPU driver image needs to be built and pushed to a Docker registry. For more details, see AMD GPU Operator / Preparing Pre-compiled Driver Images. The private registry mentioned earlier in the "target system" section can be used for this purpose.

Create a Docker registry secret in the amd-gpu namespace to enable access to the private registry. Set the <registry>, <username>, and <password> values to the information for your private registry.

kubectl create secret -n amd-gpu \
    docker-registry private-registry \
    --docker-server=<registry> \
    --docker-username=<username> \
    --docker-password=<password>

Then, create a gpu-operator-values.yaml file with the following content. Please replace <registry> with the URL of your private registry. You may also change the image name amdgpu-driver, if necessary, according to your private registry's policies.

warning

If the AMD GPU driver is already installed on the host OS, set deviceConfig.spec.driver.enable: false to skip driver installation by the GPU Operator.

info

When installing the GPU driver via the GPU Operator, refer to the User-space and kernel-space compatibility matrix to select a deviceConfig.spec.driver.version that is compatible with the host OS kernel.

gpu-operator-values.yaml

deviceConfig:
  spec:
    driver:
      enable: true
      version: "30.30.1"
      blacklist: true
      image: <registry>/amdgpu-driver
      imageRegistrySecret:
        name: private-registry
      imageRegistryTLS:
        insecure: false
        insecureSkipTLSVerify: false
      tolerations: &tolerations
        - key: amd.com/gpu
          operator: Exists
          effect: NoSchedule
    devicePlugin:
      devicePluginTolerations: *tolerations
    metricsExporter:
      prometheus:
        serviceMonitor:
          enable: true
          interval: 10s
          labels:
            release: mif
      tolerations: *tolerations

node-feature-discovery:
  enabled: false

info

• The moai-inference-framework chart already deploys Node Feature Discovery (NFD) and the moai-accelerator NodeFeatureRule for heterogeneous accelerator labeling. Set node-feature-discovery.enabled: false in the GPU Operator values to avoid installing a duplicate NFD instance. For a full list of supported hardware, see supported devices.
• If the AMD GPU Operator is already installed on your system, verify that the toleration key is set to amd.com/gpu. MoAI Inference Framework assumes this name.

You can install the AMD GPU Operator as follows.

helm upgrade -i gpu-operator rocm/gpu-operator-charts \
    --version v1.4.0 \
    -n amd-gpu \
    -f gpu-operator-values.yaml

Note that installing the operator and GPU driver may take some time. After the installation is complete, you can verify that the gpu-operator pods are running using the following command.

kubectl get pods -n amd-gpu

Expected output

NAME                                                              READY   STATUS    RESTARTS   AGE
default-device-plugin-fxj66                                       1/1     Running   0          108s
default-metrics-exporter-r2l6h                                    1/1     Running   0          108s
default-node-labeller-qhqdl                                       1/1     Running   0          2m35s
gpu-operator-gpu-operator-charts-controller-manager-69856dhd67k   1/1     Running   0          4m20s
gpu-operator-kmm-controller-7b5dd7b48b-fpcv6                      1/1     Running   0          4m20s
gpu-operator-kmm-webhook-server-c7bfc864-tfqdb                    1/1     Running   0          4m20s

info

You can monitor the installation progress using the kubectl get pods -n amd-gpu -w command instead.

---

RDMA device plugin

Host driver and OFED installation

You need to install the device drivers and OFED software for InfiniBand or RoCE NICs on the host OS. Follow the instructions provided by your hardware vendor.

This must be completed before joining the node to the Kubernetes cluster. By running the following command on the host OS, you can verify that the OFED software has been installed correctly and that it recognizes the NICs. If no devices are shown, there is an issue with the installation.

ibv_devices

Expected output

device           node GUID
<device_name>    <16-hex GUID>
<device_name>    <16-hex GUID>
...

RDMA device plugin installation

This section describes how to install the rdma-shared-device-plugin. See k8s-rdma-shared-dev-plugin / README for more details.

First, create a rdma-shared-device-plugin.yaml file as follows. You need to replace <device> with your RDMA NIC's network interface name. If multiple NICs are installed on the server, you must list all interface names (e.g., "devices": ["ib0", "ib1"]).

info

You can check the network interface names using the ip addr command.

rdma-shared-device-plugin.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rdma-devices
  namespace: kube-system
  labels:
    app.kubernetes.io/name: rdma-shared-device-plugin
    app.kubernetes.io/version: v1.5.2
    app.kubernetes.io/instance: rdma-shared-device-plugin
data:
  config.json: |
    {
      "periodicUpdateInterval": 300,
      "configList": [
        {
          "resourcePrefix": "mellanox",
          "resourceName": "hca",
          "rdmaHcaMax": 1000,
          "devices": [
            "<device>"
          ]
        }
      ]
    }

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rdma-shared-device-plugin
  namespace: kube-system
  labels:
    app.kubernetes.io/name: rdma-shared-device-plugin
    app.kubernetes.io/version: v1.5.2
    app.kubernetes.io/instance: rdma-shared-device-plugin
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: rdma-shared-device-plugin
      app.kubernetes.io/instance: rdma-shared-device-plugin
  updateStrategy:
    rollingUpdate:
      maxUnavailable: "30%"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: rdma-shared-device-plugin
        app.kubernetes.io/version: v1.5.2
        app.kubernetes.io/instance: rdma-shared-device-plugin
    spec:
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
        - key: amd.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: device-plugin
          image: ghcr.io/mellanox/k8s-rdma-shared-dev-plugin:v1.5.2
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
            - name: plugins-registry
              mountPath: /var/lib/kubelet/plugins_registry
            - name: config
              mountPath: /k8s-rdma-shared-dev-plugin
            - name: devs
              mountPath: /dev/
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
        - name: plugins-registry
          hostPath:
            path: /var/lib/kubelet/plugins_registry
        - name: config
          configMap:
            name: rdma-devices
            items:
              - key: config.json
                path: config.json
        - name: devs
          hostPath:
            path: /dev/

info

If the RDMA device plugin is already installed on your system, verify that the resource name is set to mellanox/hca. MoAI Inference Framework assumes this name. This does not imply that the actual hardware vendor must be Mellanox.

Then, create an rdma-shared-device-plugin DaemonSet using the following command.

kubectl apply -f rdma-shared-device-plugin.yaml

You can verify that the rdma-shared-device-plugin pods are running using the following command.

kubectl get pods -n kube-system -l app.kubernetes.io/instance=rdma-shared-device-plugin

Expected output

NAME                              READY   STATUS    RESTARTS   AGE
rdma-shared-device-plugin-wh9fz   1/1     Running   0          7s