LLM
Test Conditions
-
Development Board: RDK S600
-
Operating Environment: Linux
Metrics and Statistics
-
model: Name of the model.
-
BPU core num: Number of BPU (Brain Processing Unit) cores utilized during model execution.
-
qtype: Quantization precision of the model.
-
max context: Maximum cumulative token sequence length the model can handle.
-
TTFT(ms): Time To First Token; the latency for generating the first token.
-
Decode(TPS): Tokens Per Second; the generation rate during the decoding stage.
-
memory(GB): Memory usage of the model.
Measured Data
| model | BPU core num | qtype | max context | TTFT (ms) | Decode (TPS) | memory (GB) |
|---|---|---|---|---|---|---|
| DeepSeek-R1-Distill-Qwen-1.5B | prefill 2 decode 2 | prefill w4 decode w4 | 4096 | 68.9 | 92.4 | 2.2 |
| Qwen3-0.6B | prefill 4 decode 4 | prefill w8 decode w8 | 4096 | 75.4 | 92.9 | 3.0 |
| Qwen3-1.7B | prefill 4 decode 4 | prefill w4 decode w4 | 4096 | 91.2 | 75.0 | 3.7 |
| Qwen3-4B | prefill 4 decode 4 | prefill w4 decode w4 | 4096 | 232.1 | 45.8 | 6.6 |
| Qwen3-4B | prefill 4 decode 4 | prefill w8 decode w8 | 4096 | 235.3 | 32.3 | 8.3 |
| Qwen3-8B | prefill 4 decode 4 | prefill w4 decode w4 | 4096 | 283.6 | 31.4 | 9.1 |