vLLM vs Ollama: when to move to vLLM for a team

Stay on Ollama until several people are waiting on it at the same time. Ollama and llama.cpp are best for one user or a household. vLLM is a server engine built for many simultaneous requests: it batches them together on the GPU, so total throughput grows a lot as users are added, while each person's speed drops only a little. The catch is that it wants newer NVIDIA cards, more memory headroom and more setup.

The difference in one table

Both speak the OpenAI API, so apps don't care which one is behind them. You can switch later without changing your tools.

Signs you've outgrown Ollama

• People wait in line. Ollama handles a few requests in parallel, set with OLLAMA_NUM_PARALLEL, but it isn't designed for a crowd. If replies stall while a colleague's long job runs, that's the sign.
• You're running batch jobs. Thousands of documents to summarize or classify finish far faster with vLLM's batching. See batch jobs with a local model.
• You have a dedicated server. A box that runs one model all day for a team is exactly vLLM's job.

If you're one person with one card, vLLM gives you little. Single-user speed is set by memory bandwidth either way; see why bandwidth sets speed.

Hardware it needs

• A recent NVIDIA card. Ampere (RTX 30-series and the RTX A-series) or newer is the safe choice; check vLLM's docs for older generations. Pascal cards like the Tesla P100 and P40 are not supported, which is a real limit for budget racks: on P100s, llama.cpp-based tools are the option. The RTX 3090 and RTX A4500 work well.
• Headroom. vLLM claims about 90% of each card's memory by default and uses what's left after the model for a large KV cache. That cache is what lets it serve many users, so a model that barely fits is a poor choice.
• Matched cards for multi-GPU. vLLM splits a model across cards with tensor parallelism, which runs the cards together rather than taking turns. Use identical cards, in counts of 2, 4 or 8.

Starting vLLM

On Linux with an NVIDIA card and Python installed:

Then serve a model by its Hugging Face name:

That downloads the model and serves an OpenAI-compatible API on port 8000, at http://localhost:8000/v1. Useful options:

• --max-model-len caps the context. Lower it if the model won't start for lack of memory.
• --tensor-parallel-size 2 splits the model across two GPUs.
• --gpu-memory-utilization 0.8 reserves less of the card, if something else needs it.
• --api-key requires a key on every request. Set one; see securing your local AI server.

There's also an official Docker image, vllm/vllm-openai, which takes the same options and needs the same NVIDIA Container Toolkit setup as Ollama in Docker.

Which model for a team

The model sets single-user speed; the card's spare memory sets how many users fit. A mixture-of-experts model like Qwen3 30B-A3B is a strong team choice: it reads only 3B parameters per token, so it stays fast under load. A dense Qwen3 32B is smarter on hard problems but slower for everyone. Here's our calculator's single-user estimate for two RTX 3090s, 48 GB, the "Serious" build on the builds page that we pitch for small teams:

With vLLM's tensor parallelism, two matched cards can beat that single-user number, and the total across many users is far higher.

A middle path

Before vLLM, try the llama.cpp server with parallel slots (-np 4). It also batches requests, runs GGUF files, and works on older cards. It also runs on old cards like the P100s in our rack of 90 used GPUs; see serving a team on used GPUs for what that looks like.