vLLM Integration Guide

1. Introduction to vLLM

vLLM is a fast and easy-to-use library for Large Language Model (LLM) inference and serving. Developed in the Sky Computing Lab at UC Berkeley, vLLM is designed to maximize the throughput and minimize the latency of LLM serving. It achieves this through a novel attention algorithm called PagedAttention, which efficiently manages key-value caches, reducing memory waste and improving batching capabilities.

Key Features:

PagedAttention: An optimized attention algorithm that effectively manages the Key-Value cache, reducing memory overhead and improving throughput.
Continuous Batching: Processes requests in a continuous batch, maximizing GPU utilization by running multiple requests concurrently.
Optimized CUDA Kernels: Utilizes highly optimized CUDA kernels for various LLM operations, leading to faster execution.
Distributed Inference: Supports distributed inference across multiple GPUs and machines.
Compatibility: Compatible with popular LLM frameworks like Hugging Face Transformers.
OpenAI Compatible API Server: Provides an API server that mimics the OpenAI API, making it easy to integrate with existing applications.

2. Basic Installation of vLLM

vLLM can be installed via pip or from source. The recommended way is to use pip, especially if you have a compatible CUDA environment.

Prerequisites

Python 3.8+
NVIDIA GPU with CUDA 11.8 or 12.1 (for GPU acceleration)
pip package manager

Step-by-Step Installation

Option 1: Install with pip (Recommended)

This is the easiest way to get started with vLLM.

Create a Virtual Environment (Optional but Recommended) It's good practice to install Python packages in a virtual environment to avoid conflicts.
```
python3 -m venv venv_vllm
source venv_vllm/bin/activate
```
Install vLLM Install vLLM using pip. Ensure you have the correct CUDA version installed on your system. vLLM provides pre-built wheels for common CUDA versions.

For CUDA 12.1:
```
pip install vllm
```
For CUDA 11.8:
```
pip install vllm==0.3.0+cu118 -f https://storage.googleapis.com/vllm-wheels/whl/torch-2.1.0.html
```
(Note: Replace 0.3.0 with the desired vLLM version if needed. Check the vLLM GitHub releases for specific versions and their CUDA compatibility.)
Verify Installation You can verify the installation by trying to import vllm in a Python interpreter or running a quick inference example.
```
python -c "from vllm import LLM, SamplingParams; print(\'vLLM installed successfully!\')"
```

Option 2: Install from Source

This option is for developers or users who need to build vLLM for specific environments (e.g., different CUDA versions, custom modifications).

Clone the vLLM Repository

git clone https://github.com/vllm-project/vllm.git
cd vllm

Install Dependencies and Build
```
pip install -e .
```
This command installs the necessary Python dependencies and builds the CUDA kernels. This process may take some time depending on your system specifications.

Basic Usage Example

Once installed, you can use vLLM to serve LLMs. Here's a simple Python example:

from vllm import LLM, SamplingParams

# Sample prompts
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
]

# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=100)

# Create an LLM. You can specify the model name or path.
# For example, to use Llama-2-7B-Chat:
llm = LLM(model="lmsys/vicuna-7b-v1.5") # or "meta-llama/Llama-2-7b-chat-hf"

# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for prompt, output in zip(prompts, outputs):
    print(f"Prompt: {prompt!r}, Generated text: {output.outputs[0].text!r}")

This guide provides a basic overview of vLLM and its installation. For more advanced configurations, model serving, and API usage, please refer to the official vLLM documentation.

FilesExpand file tree

vllm_integration.md

Latest commit

History