Skip to main content
Version: Dev 🚧

Prefix Cache-aware Routing

Prefix caching is a technique that stores the KV cache from previous queries, allowing subsequent queries with an identical prefix to reuse it. This eliminates redundant computation and significantly improves performance for workloads with common prefixes, such as system prompts, conversation history, or shared contextual documents.

In a system with multiple inference instances (pods), each instance maintains its own (L1) prefix cache in GPU memory. Consequently, the cache hit rate varies depending on which instance a request is routed to. Prefix cache-aware routing calculates the potential cache hit rate for each pod and prioritizes routing to the pod with the highest coverage. This reduces redundant KV computation, improving both Time to First Token (TTFT) and overall throughput.

In a production environment, cache hit rates are considered alongside other factors, such as pod load and hardware characteristics, to make optimal routing decisions.

Key features​

  • Gateway integration: The AIGateway calculates cache hit rates for candidate pods and assigns scores used for routing. It stays synchronized with pod cache states through real-time ZMQ events relayed by the gateway sidecar on each inference pod.
  • SLO-based importance: The framework can dynamically weight prefix cache hits based on Service Level Objectives (SLOs) and the specific cost of KV cache recomputation for different GPU architectures.

Scorer configuration​

To enable prefix cache-aware routing, configure the prefix-cache-scorer plugin in a SchedulingProfile. The scorer tracks KV block locality across the pods bound to the gateway and scores each candidate by how many of the request's prefix blocks it already holds.

The following SchedulingProfile enables the scorer in an end-to-end (e2e) profile and selects the highest-scoring pod with max-score-picker:

prefix-cache-profile.yaml
apiVersion: heimdall.moreh.io/v1alpha1
kind: SchedulingProfile
metadata:
  name: prefix-cache
spec:
  profileHandler: e2e
  plugins:
    - type: prefix-cache-scorer
    - type: max-score-picker
  profiles:
    default:
      pluginRefs:
        - name: prefix-cache-scorer
          weight: 100
        - name: max-score-picker
kubectl apply -f prefix-cache-profile.yaml

The prefix-cache-scorer takes optional config fields for score normalization and transform — see the Plugins reference for the full table with defaults. The KV block size is not set on the scorer: it is sourced from each pod's InferenceWorker (modelCard.kvCacheBlockSize), so the scorer always uses the same block size as the engine.

Bind this SchedulingProfile to the gateway through the AIGateway's schedulingProfiles field, as shown in the deployment example below.

Components​

Token processor​

vLLM uses block hashes to look up prefixes. The scorer emulates this by hashing the request's tokens into blocks of the model's KV block size. That block size is sourced from each pod's InferenceWorker (modelCard.kvCacheBlockSize, which the sidecar reads from the engine), so the scorer's block hashes stay aligned with the engine's without any manual configuration.

KV block index​

Pods publish events (BlockStored, BlockRemoved) via ZMQ when their cache updates. The gateway sidecar on each inference pod relays these events to the gateway, which maintains a KV block index mapping prefix hashes to the list of pods holding that cache. The pod with the most matching blocks for a request is assigned the highest score.

Deployment example​

This section provides a complete example of deploying an inference service with prefix cache-aware routing using the Llama 3.2 1B Instruct model on AMD MI250 GPUs. For more details on using templates, see the Presets documentation.

1. Identify available presets​

First, find the appropriate preset for your model and hardware:

kubectl get inferenceservicetemplate -n mif \
  -l mif.moreh.io/template.type=preset \
  -l mif.moreh.io/model.name=llama-3.2-1b-instruct

2. Configure InferenceService​

The following InferenceService uses the quickstart-vllm-meta-llama-llama-3.2-1b-instruct-amd-mi250-tp2 preset. By setting ISVC_USE_KV_EVENTS: "true", the required vLLM arguments for ZMQ event publishing are automatically added.

info

In production environments, it is highly recommended to use offline hub templates (e.g., vllm-hf-hub-offline, vllm-dp-hf-hub-offline, or vllm-pp-hf-hub-offline) instead of HF_TOKEN to load pre-downloaded models from a Persistent Volume. This ensures reliability by avoiding dependencies on external network conditions during pod startup. These templates require a PVC named models in your namespace. Refer to the Hugging Face model management with persistent volume for more details.

The mif.moreh.io/aigateway label binds these pods to an AIGateway named mif. The engine's --block-size is advertised to the gateway through the pod's InferenceWorker, so the prefix-cache-scorer automatically uses the same block size — no value is set on the profile.

llama-prefix-cache.yaml
apiVersion: odin.moreh.io/v1alpha1
kind: InferenceService
metadata:
  name: llama-1b-prefix-cache
  labels:
    mif.moreh.io/aigateway: mif
spec:
  replicas: 4
  templateRefs:
    - name: vllm
    - name: quickstart-vllm-meta-llama-llama-3.2-1b-instruct-amd-mi250-tp2
    # - name: vllm-hf-hub-offline
  template:
    spec:
      containers:
        - name: main
          env:
            - name: HF_TOKEN
              value: <huggingFaceToken>
            # Enable automatic generation of --kv-events-config
            - name: ISVC_USE_KV_EVENTS
              value: "true"
            # Maintain preset defaults while specifying the block size
            - name: ISVC_EXTRA_ARGS
              value: >-
                --disable-uvicorn-access-log --no-enable-log-requests
                --max-model-len 16384 --max-num-batched-tokens 8192
                --block-size 32
info

On AMD MI250, each physical GPU is recognized as two logical devices. In this example, the preset for tp2 uses 1 physical GPU (requesting amd.com/gpu: 2).

3. Bind the profile to the gateway​

Reference the prefix-cache SchedulingProfile from the AIGateway so the gateway applies it when routing requests. Binding it to the reserved model default makes it the gateway-wide profile for every request:

aigateway.yaml
apiVersion: heimdall.moreh.io/v1alpha1
kind: AIGateway
metadata:
  name: mif
spec:
  replicas: 1
  schedulingProfiles:
    - model: default
      profile: prefix-cache

The InferenceService pods bound through the mif.moreh.io/aigateway: mif label and this AIGateway together complete the deployment. See the Quickstart for the full apply-and-verify flow.