The purpose of this benchmarking is to measure the maximum throughput (output tokens/sec) achievable when running distributed inference of the DeepSeek R1 671B model on a 5-node AMD MI300X GPU cluster. This metric directly determines the cost efficiency of inference service (tokens/$). This benchmarking demonstrates three key points:
The experimental methodology was largely designed by referring to the following report from the SGLang team, which measures the performance of PD disaggregation and expert parallelism on an NVIDIA H100 GPU cluster. The key difference is that, while the SGLang team measures prefill-only and decode-only performance separately, this benchmarking integrates prefill and decode instances and measures performance in an end-to-end inference environment, which more accurately reflects real-world achievable performance.
| Item | Description |
|---|---|
| GPU servers | 5x servers, each equipped with 8x AMD MI300X GPUs |
| Networking | InfiniBand HDR |
| Inference engine | Moreh vLLM (0.11.0rc2.moreh20251212) |
| Model | deepseek-ai/DeepSeek-R1 |
| PD disaggregation | 2x prefill, 3x decode instances |
| Parallelization | EP=8 + DP=8 |
The specifications of each GPU server are as follows:
Please make sure to install all prerequisites before starting this benchmarking. Also, please refer to the quickstart to understand how to run the MoAI Inference Framework.
In this benchmarking, you need to deploy the Istio gateway, the Heimdall scheduler configured to specify the basic routing strategy for PD disaggregation, and the Odin inference service configured to run two prefill instances and three decode instances across five GPU servers using the quickstart preset for DeepSeek R1 with DP8+EP8 on AMD MI300X.
First, you need to have a namespace for deploying and running the components of the MoAI Inference Framework. In this guide, we assume the namespace is named deepseek-r1-benchmark.
kubectl create namespace deepseek-r1-benchmark
AWS credentials must be configured in this namespace to allow the container images of the MoAI Inference Framework to be downloaded. For details, refer to the "Amazon ECR token for Moreh's container image repository" section in the prerequisites.
Then, you can use the following configuration files for the components. Click to view their contents. You must store the DeepSeek-R1 model checkpoint on the host of every worker node and specify its path in the deepseek-r1-decode.yaml and deepseek-r1-prefill.yaml files. This path will be mounted to /app/model/DeepSeek-R1 inside the pod and used to run the Moreh vLLM server.
apiVersion: v1
kind: ConfigMap
metadata:
name: mif-gateway-infrastructure
data:
service: |
spec:
type: ClusterIP
deployment: |
spec:
template:
metadata:
annotations:
proxy.istio.io/config: |
accessLogFile: /dev/stdout
accessLogEncoding: JSON
spec:
containers:
- name: istio-proxy
resources:
limits: null
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: mif
spec:
gatewayClassName: istio
infrastructure:
parametersRef:
group: ""
kind: ConfigMap
name: mif-gateway-infrastructure
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
global:
imagePullSecrets:
- name: moreh-registry
config:
apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: EndpointPickerConfig
plugins:
- type: pd-profile-handler
- type: prefill-filter
- type: decode-filter
- type: active-request-scorer
parameters:
requestTimeout: "20m"
- type: max-score-picker
schedulingProfiles:
- name: prefill
plugins:
- pluginRef: prefill-filter
- pluginRef: active-request-scorer
weight: 1
- pluginRef: max-score-picker
- name: decode
plugins:
- pluginRef: decode-filter
- pluginRef: active-request-scorer
weight: 1
- pluginRef: max-score-picker
gateway:
name: mif
gatewayClassName: istio
inferencePool:
targetPorts:
- number: 8000
- number: 8001
- number: 8002
- number: 8003
- number: 8004
- number: 8005
- number: 8006
- number: 8007
apiVersion: odin.moreh.io/v1alpha1
kind: InferenceService
metadata:
name: deepseek-r1-decode
spec:
replicas: 3
inferencePoolRefs:
- name: heimdall
templateRefs:
- name: vllm-decode-dp
- name: quickstart-vllm-deepseek-ai-deepseek-r1-decode-amd-mi300x-dp8-moe-ep8
workerTemplate:
spec:
containers:
- name: main
env:
- name: ISVC_MODEL_PATH
value: /app/model/DeepSeek-R1
- name: HF_HUB_OFFLINE
value: "1"
resources:
limits:
mellanox/hca: "1"
requests:
mellanox/hca: "1"
volumeMounts:
- name: dsr1
mountPath: /app/model/DeepSeek-R1
readOnly: false
volumes:
- name: dsr1
hostPath:
path: /path/to/DeepSeek-R1
apiVersion: odin.moreh.io/v1alpha1
kind: InferenceService
metadata:
name: deepseek-r1-prefill
spec:
replicas: 2
inferencePoolRefs:
- name: heimdall
templateRefs:
- name: vllm-prefill-dp
- name: quickstart-vllm-deepseek-ai-deepseek-r1-prefill-amd-mi300x-dp8-moe-ep8
workerTemplate:
spec:
containers:
- name: main
env:
- name: ISVC_MODEL_PATH
value: /app/model/DeepSeek-R1
- name: HF_HUB_OFFLINE
value: "1"
resources:
limits:
mellanox/hca: "1"
requests:
mellanox/hca: "1"
volumeMounts:
- name: dsr1
mountPath: /app/model/DeepSeek-R1
readOnly: false
volumes:
- name: dsr1
hostPath:
path: /path/to/DeepSeek-R1
Run the following commands to deploy and run the components.
Istio gateway:
kubectl apply -n deepseek-r1-benchmark -f gateway.yaml
kubectl get pod -n deepseek-r1-benchmark -l gateway.networking.k8s.io/gateway-name=mif
NAME READY STATUS RESTARTS AGE
mif-istio-584474ddd9-rt9p9 1/1 Running 0 163m
Heimdall scheduler:
helm upgrade -i heimdall moreh/heimdall \
--version v0.7.1 \
-n deepseek-r1-benchmark \
-f heimdall-values.yaml
kubectl get all -n deepseek-r1-benchmark -l app.kubernetes.io/instance=heimdall
NAME READY STATUS RESTARTS AGE
pod/heimdall-5576d4f48b-bgn4c 1/1 Running 0 3d1h
Odin inference service:
kubectl apply -n deepseek-r1-benchmark -f deepseek-r1-decode.yaml
kubectl apply -n deepseek-r1-benchmark -f deepseek-r1-prefill.yaml
kubectl get pod -n deepseek-r1-benchmark -l app.kubernetes.io/name=deepseek-r1-decode
kubectl get pod -n deepseek-r1-benchmark -l app.kubernetes.io/name=deepseek-r1-prefill
NAME READY STATUS RESTARTS AGE
deepseek-r1-decode-0 2/2 Running 0 5m
deepseek-r1-decode-1 2/2 Running 0 5m
deepseek-r1-decode-2 2/2 Running 0 5m
deepseek-r1-prefill-0 1/1 Running 0 5m
deepseek-r1-prefill-1 1/1 Running 0 5m
This benchmarking follows a commonly used approach for measuring the computational performance of inference servers. Multiple concurrent users send requests at a specific request-per-second (RPS) rate, each with a fixed input sequence length and output sequence length. The concurrency and RPS are determined empirically as high as possible within the limits of GPU memory capacity and without allowing requests to accumulate in the request queue of vLLM instances. The response times of these requests are measured and used to compute output tokens per second, total tokens per second, time to first token, and inter-token latency (also known as time per output token).
The experiments use the vLLM bench serve tool to conduct measurements of this kind. However, this tool was originally designed to measure the performance of a single-GPU server, and several aspects of it are insufficient for evaluating the levels of throughput observed in these experiments — tens of thousands of tokens per second. Therefore, three additional features were implemented in the vLLM bench serve tool bundled with Moreh vLLM, to correctly measure performance in a distributed inference environment with very high throughput. See the modified version here.
--warmup-time, --cooldown-time: At the beginning of the experiment, before enough requests have accumulated, and near the end of the experiment, as computation winds down, the GPUs are not fully utilized. To reliably measure the maximum throughput achievable by the inference system, the tool was extended to exclude requests from the initial (warm-up) and final (cool-down) phases from the performance measurement.--max-connections-per-worker: The tool was modified so that the response times of individual requests are recorded across multiple threads; otherwise, information for some requests may be lost.--sharegpt-input-len, --sharegpt-output-len, --gutenberg-input-len, --gutenberg-output-len: To accurately measure the effect of EP load balancing, substrings of meaningful text from a real dataset, cut to the desired input sequence length, are used as prompts rather than meaningless random strings.In this benchmarking, three different input/output sequence lengths (512/512, 1000/1000, and 2000/2000) and two different datasets (ShareGPT and Gutenberg) are evaluated. To launch a new Moreh vLLM pod in a Kubernetes cluster, first create a benchmarking-client.yaml file as follows. Please modify the following items to match your system.
ShareGPT_V3_unfiltered_cleaned_split.json file and the project_gutenberg directory on the host filesystem of that node, and specify their paths.apiVersion: v1
kind: Pod
metadata:
name: vllm-benchmark
spec:
containers:
- name: benchmark
image: 255250787067.dkr.ecr.ap-northeast-2.amazonaws.com/quickstart/moreh-vllm:vllm_251212
imagePullPolicy: IfNotPresent
command:
- sleep
args:
- infinity
volumeMounts:
- name: sharegpt-dataset
mountPath: "/app/dataset/ShareGPT_V3_unfiltered_cleaned_split.json"
- name: gutenberg-dataset
mountPath: "/app/dataset/project_gutenberg"
securityContext:
privileged: true
nodeSelector:
kubernetes.io/hostname: <clientHostname>
volumes:
- name: sharegpt-dataset
hostPath:
path: /path/to/ShareGPT_V3_unfiltered_cleaned_split.json
- name: gutenberg-dataset
hostPath:
path: /path/to/project_gutenberg
Run the following command to start the pod.
kubectl -n deepseek-r1-benchmark apply -f benchmarking-client.yaml
Inside the pod, you can run vllm bench serve. You may need to modify the --host value depending on your Istio gateway address. The following are the actual commands used to run each experiment. For each experiment, the warm-up time and cool-down time were adjusted appropriately.
vllm bench serve \
--backend vllm \
--model "deepseek-ai/DeepSeek-R1" \
--metric-percentiles "1,10,25,50,75,90" \
--percentile-metrics "itl,tps,ttft" \
--host "mif-istio.deepseek-r1-benchmark.svc.cluster.local" \
--port 80 \
--num-prompts 32400 \
--max-concurrency 10800 \
--request-rate 140 \
--ignore-eos \
--ready-check-timeout-sec 0 \
--max-connections-per-worker 1296 \
--warmup-time 120.0 \
--cooldown-time 70.0 \
--dataset-name sharegpt \
--dataset-path /app/dataset/ShareGPT_V3_unfiltered_cleaned_split.json \
--sharegpt-input-len 512 \
--sharegpt-output-len 512
vllm bench serve \
--backend vllm \
--model "deepseek-ai/DeepSeek-R1" \
--metric-percentiles "1,10,25,50,75,90" \
--percentile-metrics "itl,tps,ttft" \
--host "mif-istio.deepseek-r1-benchmark.svc.cluster.local" \
--port 80 \
--num-prompts 32400 \
--max-concurrency 10800 \
--request-rate 140 \
--ignore-eos \
--ready-check-timeout-sec 0 \
--max-connections-per-worker 1296 \
--warmup-time 130.0 \
--cooldown-time 70.0 \
--dataset-name gutenberg \
--dataset-path /app/dataset/project_gutenberg \
--gutenberg-input-len 512 \
--gutenberg-output-len 512
vllm bench serve \
--backend vllm \
--model "deepseek-ai/DeepSeek-R1" \
--metric-percentiles "1,10,25,50,75,90" \
--percentile-metrics "itl,tps,ttft" \
--host "mif-istio.deepseek-r1-benchmark.svc.cluster.local" \
--port 80 \
--num-prompts 32400 \
--max-concurrency 10800 \
--request-rate 80 \
--ignore-eos \
--ready-check-timeout-sec 0 \
--max-connections-per-worker 1296 \
--warmup-time 140.0 \
--cooldown-time 110.0 \
--dataset-name sharegpt \
--dataset-path /app/dataset/ShareGPT_V3_unfiltered_cleaned_split.json \
--sharegpt-input-len 1000 \
--sharegpt-output-len 1000
vllm bench serve \
--backend vllm \
--model "deepseek-ai/DeepSeek-R1" \
--metric-percentiles "10,25,50,75,90" \
--percentile-metrics "itl,tps,ttft" \
--host "mif-istio.deepseek-r1-benchmark.svc.cluster.local" \
--port 80 \
--num-prompts 32400 \
--max-concurrency 10800 \
--request-rate 80 \
--ignore-eos \
--ready-check-timeout-sec 0 \
--max-connections-per-worker 1296 \
--warmup-time 150.0 \
--cooldown-time 120.0 \
--dataset-name gutenberg \
--dataset-path /app/dataset/project_gutenberg \
--gutenberg-input-len 1000 \
--gutenberg-output-len 1000
vllm bench serve \
--backend vllm \
--model "deepseek-ai/DeepSeek-R1" \
--metric-percentiles "1,10,25,50,75,90" \
--percentile-metrics "itl,tps,ttft" \
--host "mif-istio.deepseek-r1-benchmark.svc.cluster.local" \
--port 80 \
--num-prompts 32400 \
--max-concurrency 10800 \
--request-rate 48 \
--ignore-eos \
--ready-check-timeout-sec 0 \
--max-connections-per-worker 1296 \
--warmup-time 300.0 \
--cooldown-time 230.0 \
--dataset-name sharegpt \
--dataset-path /app/dataset/ShareGPT_V3_unfiltered_cleaned_split.json \
--sharegpt-input-len 2000 \
--sharegpt-output-len 2000
vllm bench serve \
--backend vllm \
--model "deepseek-ai/DeepSeek-R1" \
--metric-percentiles "1,10,25,50,75,90" \
--percentile-metrics "itl,tps,ttft" \
--host "mif-istio.deepseek-r1-benchmark.svc.cluster.local" \
--port 80 \
--num-prompts 32400 \
--max-concurrency 10800 \
--request-rate 60 \
--ignore-eos \
--ready-check-timeout-sec 0 \
--max-connections-per-worker 1296 \
--warmup-time 260.0 \
--cooldown-time 240.0 \
--dataset-name gutenberg \
--dataset-path /app/dataset/project_gutenberg \
--gutenberg-input-len 2000 \
--gutenberg-output-len 2000
The results are as follows. As mentioned earlier, the concurrency and RPS values were determined empirically and may vary depending on the system scale (the number of GPU nodes). We achieved 50,892–66,194 output tokens/sec across various configurations, which corresponds to 17,000–22,000 tokens/sec per decode node.
| Input sequence length | Output sequence length | Dataset | (Concurrency, RPS) | Output tokens/sec | Output tokens/sec per decode node | Total tokens/sec | Mean TTFT (ms) | Mean ITL (ms) |
|---|---|---|---|---|---|---|---|---|
| 512 | 512 | ShareGPT | (10800, 140) | 66,194.80 | 22,064.93 | 85,347.35 | 1,677.87 | 160.33 |
| 512 | 512 | Gutenberg | (10800, 140) | 64,695.10 | 21,565.03 | 79,432.24 | 1,774.90 | 164.32 |
| 1000 | 1000 | ShareGPT | (10800, 80) | 61,828.90 | 20,609.63 | 94,103.87 | 1,802.87 | 172.16 |
| 1000 | 1000 | Gutenberg | (10800, 80) | 61,418.55 | 20,472.85 | 92,353.63 | 2,149.80 | 173.63 |
| 2000 | 2000 | ShareGPT | (10800, 48) | 51,187.87 | 17,062.62 | 77,775.33 | 2,567.87 | 208.59 |
| 2000 | 2000 | Gutenberg | (10800, 60) | 50,892.65 | 16,964.22 | 76,739.86 | 5,586.34 | 208.76 |
Click to view raw benchmarking logs.
=============Serving Benchmark Result=============
Number of worker processes: 25
Successful requests: 4333
Maximum request concurrency: 10800
Request rate configured (RPS): 140.00
Warm-up Time: 120.0
Cool-down Time: 70.0
Benchmark duration (s): 115.83
Total input tokens: 2218496
Total generated tokens: 7667539
Output token throughput (tok/s): 66194.80
Total Token throughput (tok/s): 85347.35
---------------Time to First Token----------------
Mean TTFT (ms): 1677.87
Median TTFT (ms): 1720.09
P1 TTFT (ms): 673.72
P10 TTFT (ms): 944.53
P25 TTFT (ms): 1177.09
P50 TTFT (ms): 1720.09
P75 TTFT (ms): 2117.51
P90 TTFT (ms): 2387.32
---------------Inter-token Latency----------------
Mean ITL (ms): 160.33
Median ITL (ms): 158.98
P1 ITL (ms): 105.28
P10 ITL (ms): 139.59
P25 ITL (ms): 151.42
P50 ITL (ms): 158.98
P75 ITL (ms): 169.65
P90 ITL (ms): 184.02
==================================================
=============Serving Benchmark Result=============
Number of worker processes: 25
Successful requests: 3186
Maximum request concurrency: 10800
Request rate configured (RPS): 140.00
Warm-up Time: 130.0
Cool-down Time: 70.0
Benchmark duration (s): 110.69
Total input tokens: 1631232
Total generated tokens: 7161008
Output token throughput (tok/s): 64695.10
Total Token throughput (tok/s): 79432.24
---------------Time to First Token----------------
Mean TTFT (ms): 1774.90
Median TTFT (ms): 1795.76
P1 TTFT (ms): 775.35
P10 TTFT (ms): 877.82
P25 TTFT (ms): 1124.76
P50 TTFT (ms): 1795.76
P75 TTFT (ms): 2296.75
P90 TTFT (ms): 2685.10
---------------Inter-token Latency----------------
Mean ITL (ms): 164.32
Median ITL (ms): 162.24
P1 ITL (ms): 106.99
P10 ITL (ms): 142.68
P25 ITL (ms): 154.22
P50 ITL (ms): 162.24
P75 ITL (ms): 174.62
P90 ITL (ms): 189.84
==================================================
=============Serving Benchmark Result=============
Number of worker processes: 25
Successful requests: 11856
Maximum request concurrency: 10800
Request rate configured (RPS): 80.00
Warm-up Time: 140.0
Cool-down Time: 110.0
Benchmark duration (s): 367.34
Total input tokens: 11856000
Total generated tokens: 22712445
Output token throughput (tok/s): 61828.90
Total Token throughput (tok/s): 94103.87
---------------Time to First Token----------------
Mean TTFT (ms): 1802.87
Median TTFT (ms): 1411.86
P1 TTFT (ms): 724.34
P10 TTFT (ms): 987.73
P25 TTFT (ms): 1059.29
P50 TTFT (ms): 1411.86
P75 TTFT (ms): 2221.47
P90 TTFT (ms): 3434.72
---------------Inter-token Latency----------------
Mean ITL (ms): 172.16
Median ITL (ms): 169.69
P1 ITL (ms): 120.22
P10 ITL (ms): 157.36
P25 ITL (ms): 164.97
P50 ITL (ms): 169.69
P75 ITL (ms): 177.89
P90 ITL (ms): 192.85
==================================================
=============Serving Benchmark Result=============
Number of worker processes: 25
Successful requests: 10931
Maximum request concurrency: 10800
Request rate configured (RPS): 80.00
Warm-up Time: 150.0
Cool-down Time: 120.0
Benchmark duration (s): 353.35
Total input tokens: 10931000
Total generated tokens: 21702425
Output token throughput (tok/s): 61418.55
Total Token throughput (tok/s): 92353.63
---------------Time to First Token----------------
Mean TTFT (ms): 2149.80
Median TTFT (ms): 1910.21
P10 TTFT (ms): 1040.06
P25 TTFT (ms): 1374.56
P50 TTFT (ms): 1910.21
P75 TTFT (ms): 2759.64
P90 TTFT (ms): 3502.56
---------------Inter-token Latency----------------
Mean ITL (ms): 173.63
Median ITL (ms): 171.10
P10 ITL (ms): 155.99
P25 ITL (ms): 165.99
P50 ITL (ms): 171.10
P75 ITL (ms): 180.85
P90 ITL (ms): 195.95
==================================================
=============Serving Benchmark Result=============
Number of worker processes: 25
Successful requests: 11906
Maximum request concurrency: 10800
Request rate configured (RPS): 48.00
Warm-up Time: 300.0
Cool-down Time: 230.0
Benchmark duration (s): 895.61
Total input tokens: 23812000
Total generated tokens: 45844389
Output token throughput (tok/s): 51187.87
Total Token throughput (tok/s): 77775.33
---------------Time to First Token----------------
Mean TTFT (ms): 2567.87
Median TTFT (ms): 2538.34
P1 TTFT (ms): 971.14
P10 TTFT (ms): 1213.06
P25 TTFT (ms): 1622.42
P50 TTFT (ms): 2538.34
P75 TTFT (ms): 3267.10
P90 TTFT (ms): 4126.80
---------------Inter-token Latency----------------
Mean ITL (ms): 208.59
Median ITL (ms): 201.50
P1 ITL (ms): 140.72
P10 ITL (ms): 186.91
P25 ITL (ms): 195.65
P50 ITL (ms): 201.50
P75 ITL (ms): 217.87
P90 ITL (ms): 243.87
==================================================
=============Serving Benchmark Result=============
Number of worker processes: 25
Successful requests: 12254
Maximum request concurrency: 10800
Request rate configured (RPS): 60.00
Warm-up Time: 260.0
Cool-down Time: 240.0
Benchmark duration (s): 948.19
Total input tokens: 24508000
Total generated tokens: 48255768
Output token throughput (tok/s): 50892.65
Total Token throughput (tok/s): 76739.86
---------------Time to First Token----------------
Mean TTFT (ms): 5586.34
Median TTFT (ms): 5313.19
P1 TTFT (ms): 1017.56
P10 TTFT (ms): 1745.51
P25 TTFT (ms): 2823.02
P50 TTFT (ms): 5313.19
P75 TTFT (ms): 7612.50
P90 TTFT (ms): 10096.06
---------------Inter-token Latency----------------
Mean ITL (ms): 208.76
Median ITL (ms): 201.13
P1 ITL (ms): 139.90
P10 ITL (ms): 187.16
P25 ITL (ms): 195.34
P50 ITL (ms): 201.13
P75 ITL (ms): 214.94
P90 ITL (ms): 245.98
==================================================
The following are some publicly available performance numbers for comparison.
In real production deployments, an appropriate trade-off between throughput and latency (inter-token latency and time to first token) must be chosen according to the service-level objectives (SLOs). As shorter latency targets are pursued, achievable throughput inevitably decreases. Nevertheless, measuring and comparing the maximum achievable throughput before applying SLO constraints is an important step in evaluating infrastructure efficiency. Our next benchmarking will examine how throughput varies across different ITL targets.
An input sequence length of 2,000 and an output sequence length of 100 were first used by the SGLang team for their PD+EP performance evaluation. Since then, this configuration has been widely adopted to evaluate PD+EP performance of DeepSeek R1.
First, please note that this configuration was proposed to measure prefill and decode throughput separately. Under the assumption that the input length is always 20x longer, a real inference system would require ~10x more prefill instances than decode instances. (In practice, real usage patterns differ from this assumption, and the number of decode instances typically exceeds that of prefill instances.) In small clusters, prefill inevitably becomes the overall performance bottleneck, making it impossible to accurately measure the output tokens/sec that the GPU servers can actually deliver.
Despite this, by enabling prefix caching and having input sequences share a fixed set of prompts, we can design a scenario in which the prefill workload is significantly reduced and measure the resulting output tokens/sec. As a result, we achieved ~18,000 tokens/sec per decode node.
| Input sequence length | Output sequence length | Dataset | (Concurrency, RPS) | Output tokens/sec | Output tokens/sec per decode node | Mean ITL (ms) |
|---|---|---|---|---|---|---|
| 2000 | 100 | Gutenberg | (10800, 1500) | 53,776.62 | 17,925.54 | 191.71 |
Click to view the raw benchmarking log.
=============Serving Benchmark Result=============
Number of worker processes: 30
Successful requests: 185577
Maximum request concurrency: 10800
Request rate configured (RPS): 1500.00
Warm-up Time: 30.0
Cool-down Time: 20.0
Benchmark duration (s): 367.68
Total input tokens: 371154000
Total generated tokens: 19772555
Output token throughput (tok/s): 53776.62
Total Token throughput (tok/s): 1063226.66
---------------Time to First Token----------------
Mean TTFT (ms): 1079.36
Median TTFT (ms): 979.51
P10 TTFT (ms): 832.05
P25 TTFT (ms): 897.71
P50 TTFT (ms): 979.51
P75 TTFT (ms): 1079.35
P90 TTFT (ms): 1223.28
---------------Inter-token Latency----------------
Mean ITL (ms): 191.71
Median ITL (ms): 181.35
P10 ITL (ms): 166.87
P25 ITL (ms): 175.00
P50 ITL (ms): 181.35
P75 ITL (ms): 191.88
P90 ITL (ms): 259.46
==================================================
We have also measured the performance of a decoding-only execution under the same configuration (ISL=2,000, OSL=100) and reported the results in a technical report. The maximum throughput achieved in this setting was 21,224 tokens/sec per decode node. This indicates that, in an end-to-end environment, MoAI Inference Framework is able to achieve ~85% of the peak decode performance.
]]>