Skip to main content
Version: Dev 🚧

Supported devices

Accelerator labels​

We believe that the types of accelerators used to build AI data centers will continue to diversify, and the MoAI Inference Framework provides features to efficiently utilize clusters with heterogeneous accelerators.

Each accelerator is identified by labels prefixed with moai.moreh.io. Users can either specify the accelerator on which a model should run, or allow the framework to automatically select the optimal accelerator. Below is the list of accelerator labels supported by the MoAI Inference Framework.

LabelAccelerator
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi355x		AMD Instinct MI355X GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi350x		AMD Instinct MI350X GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi325x		AMD Instinct MI325X GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi308x		AMD Instinct MI308X GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi300x		AMD Instinct MI300X GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi300x-hf		AMD Instinct MI300X HF GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi300a		AMD Instinct MI300A APU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi250x		Half of AMD Instinct MI250X GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi250		Half of AMD Instinct MI250 GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi210		AMD Instinct MI210 GPU
moai.moreh.io/accelerator.vendor: amd
moai.moreh.io/accelerator.model: mi100		AMD Instinct MI100 GPU
moai.moreh.io/accelerator.vendor: nvidia
moai.moreh.io/accelerator.model: h100		NVIDIA H100 SXM GPU
moai.moreh.io/accelerator.vendor: nvidia
moai.moreh.io/accelerator.model: h100-pcie		NVIDIA H100 PCIe GPU
moai.moreh.io/accelerator.vendor: nvidia
moai.moreh.io/accelerator.model: h100-80gb-hbm3		NVIDIA H100 80GB HBM3 GPU
moai.moreh.io/accelerator.vendor: nvidia
moai.moreh.io/accelerator.model: h800		NVIDIA H800 GPU
moai.moreh.io/accelerator.vendor: nvidia
moai.moreh.io/accelerator.model: h800-pcie		NVIDIA H800 PCIe GPU
moai.moreh.io/accelerator.vendor: nvidia
moai.moreh.io/accelerator.model: a100-80gb-pcie		NVIDIA A100 80GB PCIe GPU
moai.moreh.io/accelerator.vendor: nvidia
moai.moreh.io/accelerator.model: a100-40gb-pcie		NVIDIA A100 40GB PCIe GPU
moai.moreh.io/accelerator.vendor: tenstorrent
moai.moreh.io/accelerator.model: blackhole		Tenstorrent Blackhole NPU
moai.moreh.io/accelerator.vendor: tenstorrent
moai.moreh.io/accelerator.model: wormhole		Tenstorrent Wormhole NPU

Checking available accelerators​

You can use the following command to see which accelerators are currently available in the cluster and how they are labeled by the MoAI Inference Framework.

kubectl get nodes -L moai.moreh.io/accelerator.vendor,moai.moreh.io/accelerator.model
NAME         STATUS     ROLES           AGE    VERSION   ACCELERATOR.VENDOR   ACCELERATOR.MODEL
cpu-001      Ready      control-plane   1d     v1.32.5
mi250-001    Ready      <none>          1d     v1.32.5   amd                  mi250
mi300x-001   Ready      <none>          1d     v1.32.5   amd                  mi300x

To list nodes that match a specific label, use the -l option:

kubectl get nodes -l moai.moreh.io/accelerator.model=mi300x
NAME         STATUS   ROLES    AGE   VERSION
mi300x-001   Ready    <none>   1d   v1.32.5