This page provides a high-level introduction to vLLM's architecture, core components, and design principles. It serves as an entry point for understanding how vLLM orchestrates large language model inference across multiple hardware platforms with optimized memory management and execution.
vLLM V1 represents a significant re-architecture of the core engine (scheduler, KV cache manager, worker, and sampler) to provide a cohesive, modular, and high-performance framework while retaining the stable model implementations and kernels from V0 docs/usage/v1_guide.md9-15
vLLM is a fast and easy-to-use library for LLM inference and serving README.md24 It optimizes throughput and memory efficiency through several key technologies:
Sources: README.md24-62 docs/usage/v1_guide.md9-39
vLLM follows a layered architecture with a clear separation of concerns. The V1 engine introduces a decoupled execution model where the EngineCore runs in a separate process from the API frontend to minimize CPU overhead and Python GIL contention docs/usage/v1_guide.md19-20 vllm/v1/engine/core_client.py84-105
Title: "vLLM System Architecture (Natural Language to Code Entities)"
Sources: vllm/engine/arg_utils.py35-120 vllm/config/vllm.py30-52 vllm/config/model.py107-170 vllm/v1/engine/core.py96-158 vllm/v1/engine/core_client.py71-132 vllm/v1/engine/input_processor.py36-144 vllm/v1/engine/output_processor.py36-102 vllm/v1/core/sched/interface.py54 vllm/v1/kv_cache_interface.py80 vllm/v1/executor/__init__.py79 vllm/v1/worker/worker_base.py41 vllm/v1/worker/gpu_model_runner.py218
Layered Architecture Overview
| Layer | Purpose | Key Components |
|---|---|---|
| External Interface | Entry points for users | LLM, AsyncLLM, OpenAI API server vllm/entrypoints/llm.py66 vllm/v1/engine/async_llm.py71 vllm/entrypoints/openai/api_server.py117 |
| Configuration | Argument parsing and config assembly | EngineArgs, VllmConfig, ModelConfig, ParallelConfig vllm/v1/engine/core.py24 vllm/engine/arg_utils.py35-120 |
| Engine Orchestration | Request lifecycle and IPC coordination | EngineCore, InputProcessor, OutputProcessor vllm/v1/engine/core.py103 vllm/v1/engine/input_processor.py36 vllm/v1/engine/async_llm.py136 |
| Scheduling & Memory | Resource allocation and KV management | SchedulerInterface, KVCacheConfig vllm/v1/core/sched/interface.py55 vllm/v1/kv_cache_interface.py86 |
| Execution | Model forward passes on hardware | Executor, Worker, GPUModelRunner vllm/v1/executor/__init__.py80 vllm/v1/worker/gpu_worker.py126 vllm/v1/worker/gpu_model_runner.py218 |
Sources: vllm/v1/engine/core.py96-158 vllm/v1/engine/async_llm.py71-154 vllm/entrypoints/llm.py66-162 vllm/v1/engine/core_client.py71-132 vllm/engine/arg_utils.py35-120
EngineCore is the high-performance inner loop of vLLM. It manages the Scheduler and the Executor vllm/v1/engine/core.py103-130 It initializes specialized configurations and hardware-specific optimizations.
Key responsibilities:
_initialize_kv_caches, and sets up the model executor and scheduler vllm/v1/engine/core.py130-166EngineCoreRequest objects containing prompt token IDs, multimodal features, and sampling parameters vllm/v1/engine/core.py64 vllm/v1/engine/__init__.py88-146Sources: vllm/v1/engine/core.py103-186
EngineCoreClient abstracts the communication between the frontend (API) and the backend (EngineCore) vllm/v1/engine/core_client.py71:
EngineCore in the same process as the caller, typically for offline inference vllm/v1/engine/core_client.py105AsyncLLM vllm/v1/engine/core_client.py132Sources: vllm/v1/engine/core_client.py71-132 vllm/v1/engine/core.py103-186
The flow below demonstrates how a user request traverses the system from a high-level API call to GPU execution.
Title: "vLLM V1 Request Lifecycle (Code Entity Space)"
Sources: vllm/v1/engine/async_llm.py136-144 vllm/v1/engine/core_client.py143-212 vllm/v1/engine/core.py158-166 vllm/v1/engine/core.py130 vllm/v1/worker/gpu_model_runner.py465-470
Request Lifecycle Stages:
InputProcessor validates SamplingParams or PoolingParams and converts EngineInput into EngineCoreRequest vllm/v1/engine/input_processor.py36-144 vllm/v1/engine/async_llm.py136Scheduler determines which requests enter the current batch based on token budgets and cache availability vllm/v1/engine/core.py158-166Executor coordinates workers to run the model forward pass. On GPUs, this involves GPUModelRunner managing the InputBatch and kernel dispatch vllm/v1/worker/gpu_model_runner.py465OutputProcessor collects EngineCoreOutputs, updates request states, and handles detokenization vllm/v1/engine/async_llm.py139-144Sources: vllm/v1/engine/core.py103-186 vllm/v1/engine/async_llm.py133-154 vllm/v1/engine/core_client.py143-212 vllm/v1/worker/gpu_model_runner.py465-470
vLLM V1 is designed for extreme scale and performance:
CoreEngineProcManager which handles creation, readiness, and shutdown vllm/v1/engine/utils.py120-171ElasticEPScalingExecutor vllm/v1/worker/gpu_worker.py149StructuredOutputManager and grammar-based generation support vllm/v1/engine/core.py142torch.compile and CUDA graph capture modes (CUDAGraphMode) to minimize kernel launch overhead vllm/config/vllm.py81-94 vllm/v1/worker/gpu_model_runner.py24-30StatLoggerManager, providing snapshots of SchedulerStats and IterationStats vllm/v1/metrics/loggers.py158-166 vllm/v1/metrics/stats.py28-100Sources: vllm/v1/engine/utils.py120-171 vllm/v1/engine/core_client.py126-132 vllm/v1/worker/gpu_worker.py149 vllm/v1/engine/core.py142 vllm/v1/metrics/loggers.py158-166 vllm/config/vllm.py81-94
Refresh this wiki
This wiki was recently refreshed. Please wait 3 days to refresh again.