Supported devices
Accelerator labels​
The types of accelerators used to build AI data centers will continue to diversify, and the MoAI Inference Framework provides features to efficiently utilize clusters with heterogeneous accelerators.
Each accelerator is identified by labels prefixed with moai.moreh.io. Users can either specify the accelerator on which a model should run, or allow the framework to automatically select the optimal accelerator. Below is the list of accelerator labels supported by the MoAI Inference Framework.
| Label | Accelerator |
|---|---|
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi355x | AMD Instinct MI355X GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi350x | AMD Instinct MI350X GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi325x | AMD Instinct MI325X GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi308x | AMD Instinct MI308X GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi300x | AMD Instinct MI300X GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi300x-hf | AMD Instinct MI300X HF GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi300a | AMD Instinct MI300A APU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi250x | Half of AMD Instinct MI250X GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi250 | Half of AMD Instinct MI250 GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi210 | AMD Instinct MI210 GPU |
moai.moreh.io/accelerator.vendor: amdmoai.moreh.io/accelerator.model: mi100 | AMD Instinct MI100 GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: b300 | NVIDIA B300 GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: b200 | NVIDIA B200 GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: h200-sxm | NVIDIA H200 SXM GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: h100-pcie | NVIDIA H100 PCIe GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: h100-sxm | NVIDIA H100 SXM GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: h100-nvl | NVIDIA H100 NVL GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: h20-3e | NVIDIA H20-3e GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: h800 | NVIDIA H800 GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: h800-pcie | NVIDIA H800 PCIe GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: l40s | NVIDIA L40S GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: a100-80gb-sxm | NVIDIA A100 80GB SXM GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: a100-80gb-pcie | NVIDIA A100 80GB PCIe GPU |
moai.moreh.io/accelerator.vendor: nvidiamoai.moreh.io/accelerator.model: a100-40gb-pcie | NVIDIA A100 40GB PCIe GPU |
moai.moreh.io/accelerator.vendor: tenstorrentmoai.moreh.io/accelerator.model: blackhole | Tenstorrent Blackhole NPU |
moai.moreh.io/accelerator.vendor: tenstorrentmoai.moreh.io/accelerator.model: wormhole | Tenstorrent Wormhole NPU |
Checking available accelerators​
You can use the following command to see which accelerators are currently available in the cluster and how they are labeled by the MoAI Inference Framework.
kubectl get nodes -L moai.moreh.io/accelerator.vendor,moai.moreh.io/accelerator.model
Expected output
NAME STATUS ROLES AGE VERSION ACCELERATOR.VENDOR ACCELERATOR.MODEL
cpu-001 Ready control-plane 1d v1.32.5
mi250-001 Ready <none> 1d v1.32.5 amd mi250
mi300x-001 Ready <none> 1d v1.32.5 amd mi300x
To list nodes that match a specific label, use the -l option:
kubectl get nodes -l moai.moreh.io/accelerator.model=mi300x
Expected output
NAME STATUS ROLES AGE VERSION
mi300x-001 Ready <none> 1d v1.32.5