LLM Inference Metrics: TPS, TTFT, and What They Mean for Performance

Mar 27, 2026

I was staring at my terminal, waiting for a response from a locally-run LLM. The clock ticked: 5 seconds, 10 seconds, 30 seconds. When it finally started generating, I wondered: was this slow? Fast? How do I even measure this?

I asked around and kept hearing two acronyms: TPS and TTFT. But what do they actually mean?

The Confusion

I found a Reddit thread where someone was running Qwen on old hardware and mentioned “Decode TPS? TTFT?” with logs showing tokenization rates of 272,829 tok/s. The post mentioned 25-minute wait times for responses. The comments revealed the real issue: those metrics tell very different parts of the performance story.

Let me break down what I learned.

TTFT: Time To First Token

TTFT measures the latency from when you send a request to when you see the first character output.

Request sent ──────────────────────> First token appears
                  TTFT = this duration

This metric matters most for:

Chat interfaces where users expect quick responses
Interactive applications like coding assistants
Any scenario where perceived responsiveness is critical

What affects TTFT:

Prompt length (longer prompts = more prefill work)
Model size (larger models = more computation)
KV cache efficiency
Hardware memory bandwidth

Rule of thumb: Under 500ms feels responsive. Over 2 seconds feels sluggish.

TPS: Tokens Per Second

TPS measures how fast the model generates text after that first token.

Total tokens generated: 500
Total generation time: 25 seconds
TPS = 500 / 25 = 20 tokens/second

This metric matters most for:

Long-form content generation
Batch processing workloads
Throughput optimization

What affects TPS:

Model architecture and size
GPU/CPU capabilities
Batch size (larger batches can improve throughput)
Decoding strategy (sampling vs greedy)
Memory bandwidth

Rule of thumb: 20+ TPS reads smoothly. Under 10 TPS feels noticeably slow.

The Trade-off

Here’s where it gets interesting. These metrics often trade off against each other:

High TTFT + High TPS  =  Slow start, fast finish
                          Good for: batch jobs, long documents

Low TTFT + Low TPS    =  Quick start, slow generation
                          Good for: chat, interactive use

Low TTFT + High TPS   =  The ideal (requires good hardware)
                          Good for: everything

Measuring These Metrics

I wrote a simple Python script to measure both:

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def measure_inference(model_name, prompt):
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.float16,
        device_map="auto"
    )

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    # Measure everything
    start_prefill = time.time()

    # Generate with timing hooks
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        return_dict_in_generate=True,
        output_scores=True
    )

    end_time = time.time()

    # Calculate metrics
    total_tokens = outputs.sequences.shape[1] - inputs.input_ids.shape[1]
    total_time = end_time - start_prefill

    # Note: True TTFT requires streaming callbacks
    # This is a simplified measurement
    return {
        "total_time": total_time,
        "total_tokens": total_tokens,
        "avg_tps": total_tokens / total_time if total_time > 0 else 0
    }

# Usage
result = measure_inference("meta-llama/Llama-2-7b-hf", "Explain quantum computing")
print(f"Average TPS: {result['avg_tps']:.2f}")

For more accurate measurements, vLLM provides built-in benchmarking:

# Start the server
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-hf \
    --enable-prefix-caching

# In another terminal, run benchmarks
python benchmarks/benchmark_serving.py \
    --model meta-llama/Llama-2-7b-hf \
    --num-prompts 100 \
    --request-rate 10

Common Mistakes I Made

Mistake 1: Only looking at one metric

I initially focused only on TPS because it seemed like the obvious speed measure. But for my chat application, TTFT was actually more important for user experience.

Mistake 2: Ignoring prompt processing

That tokenization rate of 272,829 tok/s I mentioned earlier? That’s just input processing speed. It doesn’t tell you anything about generation speed.

Input processing: 272,829 tok/s  (very fast, CPU-bound)
Output generation: 5-20 tok/s    (much slower, GPU/memory-bound)

Mistake 3: Not considering batch effects

Single-request TPS looks different from batch TPS. When serving multiple users, batching can dramatically improve throughput but might increase individual TTFT.

Optimization Techniques

After understanding these metrics, optimization becomes clearer:

For TTFT:

Use KV caching for repeated prompts
Implement prefix caching
Consider speculative decoding

For TPS:

Use flash attention
Optimize batch size
Consider quantization (4-bit, 8-bit)

from vllm import LLM, SamplingParams

# vLLM handles both metrics automatically
llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    enable_prefix_caching=True,  # Improves TTFT
    gpu_memory_utilization=0.9,   # Maximizes throughput
)

sampling_params = SamplingParams(
    max_tokens=100,
    temperature=0.7
)

outputs = llm.generate(["Your prompt here"], sampling_params)

When to Prioritize Which Metric

┌─────────────────────────┬────────────┬─────────────┐
│ Use Case                │ Priority  │ Target     │
├─────────────────────────┼────────────┼─────────────┤
│ Chat bot                │ Low TTFT   │ < 500ms    │
│ Code completion         │ Low TTFT   │ < 200ms    │
│ Document summarization  │ High TPS   │ > 20 TPS   │
│ Batch processing        │ High TPS   │ Maximize   │
│ Real-time assistant     │ Both       │ Balance    │
└─────────────────────────┴────────────┴─────────────┘

The Real-World Impact

Going back to that Reddit post with 25-minute wait times: the issue wasn’t just slow TPS. The extreme TTFT meant the user had no feedback for 25 minutes. Even a “thinking…” indicator would have improved the experience.

Understanding these metrics helped me:

Choose the right hardware for my use case
Set realistic expectations for performance
Know what to optimize first
Compare different models fairly

Key Takeaways

TTFT = user experience quality (how fast it starts)
TPS = throughput efficiency (how fast it generates)
Measure both for a complete performance picture
Optimize based on your specific use case

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 vLLM Documentation
👨‍💻 HuggingFace Transformers Performance
👨‍💻 llama.cpp Benchmarking

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!