Cutting my RAG assistant's first word from 50 seconds to 4

I added an assistant to my portfolio where visitors could ask questions about me: "Ask Göktuğ". The architecture is classic RAG: when a question arrives, the knowledge base is searched first, the passages found are given to a language model, and the model writes an answer based only on those passages.

My first goal was to run everything on my own server: the search and the language model that writes the answer. The server is a shared machine with 4 CPU cores, 8 GB of memory, no GPU, and other sites on it. For the language model I picked a small 3-billion-parameter model (qwen2.5:3b) running on Ollama.

The assistant answered correctly, but the measurements looked like this:

• For a new question the first word arrived in 20–25 seconds; if the model had been unloaded from memory, it went up to ~60 seconds.
• Answers streamed at 5–6.6 tokens per second.
• With the model loaded, Ollama held about 3.1 GB of memory.

A real visitor's first question timed out after waiting 56 seconds. A portfolio visitor will not wait a minute; if nothing shows up within a few seconds, they close the tab.

There was one more surprise along the way: the model's prompt cache kept growing toward 8 GB and hit the 4 GB memory limit I had set on the service. Capping the cache with LLAMA_ARG_CACHE_RAM=256 fixed that. But the real problem remained: on this CPU, a 3-billion-parameter model was too slow for chat.

I split the system into two parts and looked at each separately:

1. Search (embedding + BM25): light, fast and tightly tied to my knowledge base. No problem keeping it on the server.
2. Answer generation (the language model): heavy and slow on a CPU. This was the bottleneck.

So I moved only answer generation to a cloud language model (MiniMax) through an OpenAI-compatible API. The only thing that changed in the code was the provider layer: one setting in the .env file switched back to the local model. Search and the embedding model stayed on the server, and I kept the embedding model in memory permanently (~0.8 GB) so it would not reload on every question.

The switch needed one small fix: the model I used writes a "thinking" text before its answer. I wrote a filter that strips that part out of the stream, so visitors see only the answer.

• The first word dropped to ~4 seconds on average.
• Answers started streaming at ~54 tokens per second.
• The 40-question accuracy test passed 40/40 right after the switch; accuracy did not drop.

There is a cost: visitor questions now go to an external service, and every question costs a little. I kept that under control, against both abuse and cost, with per-visitor and daily global question limits.

Even after the speed-up, the assistant still held about 1 GB of memory on the server, and on a shared server that is a share taken from everything else. After a while I decided to shut it down. The code and the knowledge base are still there; bringing it back when resources allow is a single command.

• Measure first. "Local models are slow" is a guess; "the first word takes 25 seconds" is data, and it makes the decision easy.
• Break the system into parts. There was no need to move everything to the cloud; moving the bottleneck was enough.
• Keep the decision reversible. Tying the provider to a single setting made both trying it and going back cheap.
• Re-measure accuracy after every change. The accuracy test, not me, said I had gained speed without losing quality.