Skip to main content
Version: Dev 🚧

Prefill-Decode Disaggregation

During LLM inference, computation occurs in two stages: prefill and decode. In the prefill phase, the model processes the entire input prompt to generate the first token — a highly parallel, compute-bound process. The decode phase then predicts one token at a time, reusing the growing KV cache, and is memory-bound.

Because these phases have fundamentally different characteristics, prefill-decode (PD) disaggregation executes them on separate GPU resources. The prefill runs first on compute-optimized machines, then the KV cache is transferred to memory-optimized ones for decoding. This separation allows each phase to use its optimal parallelization, batch size, and configurations, while preventing interference between concurrent requests.

PD disaggregation can improve key metrics such as time to first token (TTFT) and time per output token (TPOT) — since TTFT depends on prefill and TPOT on decode, dedicated optimization for each leads to better overall performance. However, because it also introduces communication overhead, which may negatively affect TTFT, PD disaggregation should be applied judiciously to ensure net efficiency gains.

Key features​

  • The AIGateway routes requests across prefill-only and decode-only instances, allowing each phase to scale independently while managing the prefill-to-decode handoff for every request.
  • The framework can automatically determine whether to apply PD disaggregation and how to scale each phase according to defined service level objectives (SLOs).
  • Moreh vLLM is optimized to efficiently execute both prefill and decode phases of various models on AMD MI200 and MI300 series GPUs. It applies distinct parallelization and optimization strategies tailored to prefill-only and decode-only instances.

Configuration​

info

This section covers only the differences from the standard deployment described in the Quickstart. Read the Quickstart first to understand the base setup.

To enable PD disaggregation, you configure a pd SchedulingProfile (to route prefill and decode separately) and deploy prefill and decode InferenceServices with the correct roles and hardware.

Scheduling profile​

Create a SchedulingProfile with profileHandler: pd. Unlike the end-to-end (e2e) handler, the pd handler runs two sub-profiles, prefill and decode. For each request the gateway runs both sub-pipelines, picks a prefill pod and a decode pod, and disaggregates the request across them; if only one role has available pods, it falls back to serving the request end-to-end on that pick.

Each sub-profile lists only the scorers and a picker. You do not add a role filter: the gateway automatically restricts the prefill sub-profile to pods labeled mif.moreh.io/role: prefill and the decode sub-profile to pods labeled mif.moreh.io/role: decode. Those labels are set by the prefill/decode presets and runtime-bases, so you do not set them manually (see InferenceService configuration below).

pd-profile.yaml
apiVersion: heimdall.moreh.io/v1alpha1
kind: SchedulingProfile
metadata:
  name: pd
spec:
  profileHandler: pd
  plugins:
    - type: waiting-requests-scorer
    - type: max-score-picker
  profiles:
    prefill:
      pluginRefs:
        - name: waiting-requests-scorer
          weight: 100
        - name: max-score-picker
    decode:
      pluginRefs:
        - name: waiting-requests-scorer
          weight: 100
        - name: max-score-picker
kubectl apply -f pd-profile.yaml

Bind the profile to the gateway through the AIGateway's schedulingProfiles field. Binding it to the reserved model default applies it to every request:

aigateway.yaml
apiVersion: heimdall.moreh.io/v1alpha1
kind: AIGateway
metadata:
  name: mif
spec:
  replicas: 1
  schedulingProfiles:
    - model: default
      profile: pd

InferenceService configuration with presets​

For the InferenceService, use presets (InferenceServiceTemplate) to simplify the configuration of hardware-specific settings (like parallelism strategy) for different GPU types. For more details on presets, see the Presets documentation.

warning

PD disaggregation requires high-bandwidth network connectivity (e.g., RDMA) between prefill and decode pods for KV cache transfer. The specific network resource configuration (such as mellanox/hca) varies by cluster. Consult your cluster administrator to determine the correct network resource type and limits for your environment.

List available prefill and decode presets for a specific model:

kubectl get inferenceservicetemplate -n mif \
  -l mif.moreh.io/template.type=preset \
  -l mif.moreh.io/model.name=deepseek-r1

In this example, DeepSeek-R1 is deployed using:

  • Prefill: AMD MI300X (DP=8, Expert Parallel)
  • Decode: AMD MI308X (DP=8, Expert Parallel)

Two InferenceService resources are created. Because the presets use data parallelism (dp8-moe-ep8), the runtime-bases are vllm-prefill-dp and vllm-decode-dp, and the ISVC uses workerTemplate instead of template. Both are bound to the same AIGateway (named mif above) through the mif.moreh.io/aigateway label.

info

The prefill and decode runtime-bases stamp the mif.moreh.io/role label (prefill or decode) onto their pods automatically. The gateway's pd profile reads this label to route each phase to the right pods, so you do not set the role yourself — choosing a vllm-prefill* or vllm-decode* runtime-base is what assigns the role.

info

Choose the appropriate PD-specific runtime-base template according to your parallelism strategy:

  • TP-only (Tensor Parallel): Use vllm-prefill and vllm-decode.
  • DP (Data Parallel): Use vllm-prefill-dp and vllm-decode-dp.
  • PP (Pipeline Parallel): Use vllm-prefill-pp and vllm-decode-pp.

In production environments, it is highly recommended to use offline hub templates (e.g., vllm-hf-hub-offline, vllm-dp-hf-hub-offline, or vllm-pp-hf-hub-offline) instead of HF_TOKEN to load pre-downloaded models from a Persistent Volume. This ensures reliability by avoiding dependencies on external network conditions during pod startup. These templates require a PVC named models in your namespace. Refer to the Hugging Face model management with persistent volume for more details.

1. Prefill Service (MI300X)​

This service uses the MI300X prefill preset and the vllm-prefill-dp runtime-base for data-parallel prefill.

deepseek-r1-prefill.yaml
apiVersion: odin.moreh.io/v1alpha1
kind: InferenceService
metadata:
  name: deepseek-r1-prefill
  labels:
    mif.moreh.io/aigateway: mif
spec:
  templateRefs:
    # 1. Runtime base for data-parallel prefill
    - name: vllm-prefill-dp
    # 2. Preset for DeepSeek-R1 on MI300X with DP=8 and Expert Parallel
    - name: moreh-vllm-0.15.0-260226-rc2-deepseek-ai-deepseek-r1-prefill-amd-mi300x-dp8-moe-ep8
  workerTemplate:
    spec:
      containers:
        - name: main
          env:
            - name: HF_TOKEN
              value: <huggingFaceToken>
          # 3. Specify cluster-specific hardware resources (e.g., RDMA NICs for KV cache transfer)
          resources:
            limits:
              mellanox/hca: "1"

2. Decode Service (MI308X)​

This service uses the MI308X decode preset and the vllm-decode-dp runtime-base for data-parallel decode.

deepseek-r1-decode.yaml
apiVersion: odin.moreh.io/v1alpha1
kind: InferenceService
metadata:
  name: deepseek-r1-decode
  labels:
    mif.moreh.io/aigateway: mif
spec:
  templateRefs:
    # 1. Runtime base for data-parallel decode
    - name: vllm-decode-dp
    # 2. Decode preset for DeepSeek-R1 on MI308X with DP=8 and Expert Parallel
    - name: moreh-vllm-0.15.0-260226-rc2-deepseek-ai-deepseek-r1-decode-amd-mi308x-dp8-moe-ep8
  workerTemplate:
    spec:
      containers:
        - name: main
          env:
            - name: HF_TOKEN
              value: <huggingFaceToken>
          # 3. Specify cluster-specific hardware resources (e.g., RDMA NICs for KV cache transfer)
          resources:
            limits:
              mellanox/hca: "1"