Fine‑Tuning Local LLMs with Ollama on a $5 VPS: Complete Tutorial Using OpenWebUI

Running a customized large language model (LLM) locally used to feel like a luxury reserved for those with beefy workstations or deep pockets for cloud GPU instances. Today, thanks to projects like Ollama and OpenWebUI, you can pull a base model, fine‑tune it with your own data, and expose it through a clean web interface—all on a modest $5/month VPS. This tutorial walks you through every step, from provisioning the server to testing the final chatbot, while keeping expectations realistic about what you can achieve on limited hardware.

Prerequisites#

Before you begin, make sure you have:

• A Linux VPS with at least 1 GB RAM and 25 GB SSD (the cheapest $5 plans from providers like DigitalOcean, Linode, or Vultr usually meet this).
• Root or sudo access.
• Basic familiarity with the Linux command line (ssh, apt, systemctl).
• A small dataset for fine‑tuning (e.g., a few hundred lines of FAQs, support tickets, or domain‑specific text). We’ll use a simple JSONL format.

Honesty note: Fine‑tuning a 7B‑parameter model on a 1 GB RAM VPS is not feasible. We’ll work with a smaller model (e.g., `phi‑2, TinyLlama, or a 1.3B variant) that fits in memory and can be adapted with lightweight techniques like LoRA. Expect modest gains rather than state‑of‑the‑art performance.

Step 1: Provision the VPS#

1. Create the instance – Choose Ubuntu 22.04 LTS (the most compatible with Ollama’s binaries).
2. Update the system:

   sudo apt update && sudo apt upgrade -y
   

3. Install essential tools:

   sudo apt install -y curl wget git vim
   

4. Set a hostname (optional but helpful):

   sudo hostnamectl set-hostname ollama-vps
   

Step 2: Install Ollama#

Ollama provides a single‑binary installer that sets up a systemd service.

# Download and run the install script
curl -fsSL https://ollama.com/install.sh | sh

The script adds the ollama user, installs the binary to /usr/local/bin/ollama, and enables a service.

Check the status:

systemctl status ollama
# Should show active (running)

If the service fails to start, look at the logs:

journalctl -u ollama -f

Step 3: Pull a Base Model#

We need a model small enough to run on limited RAM yet capable of being fine‑tuned. TinyLlama‑1.1B‑Chat is a good candidate (~1.3 GB). Pull it with:

ollama pull tinyllama

Verify the model is available:

ollama list
# Should show tinyllama among others

You can test a quick inference to ensure everything works:

ollama run tinyllama "Hello, who are you?"

Expect a response within a few seconds. If it’s painfully slow or OOM‑killed, consider an even smaller model like phi-2 or starling-lm-7b-alpha (though the latter is larger). Adjust accordingly.

Step 4: Prepare Custom Data for Fine‑Tuning#

Ollama’s fine‑tuning workflow uses the LoRA (Low‑Rank Adaptation) method via the ollama create command, which expects a Modelfile and a training dataset in JSONL format.

4.1 Create a training directory#

mkdir -p ~/ollama-finetune/data
cd ~/ollama-finetune

4.2 Example JSONL file#

Each line is a JSON object with prompt and response fields. For a simple FAQ:

{"prompt":"What is the refund policy?","response":"You can request a refund within 30 days of purchase."}
{"prompt":"How do I reset my password?","response":"Visit the login page and click 'Forgot Password'."}
{"prompt":"What are your support hours?","response":"Support is available 24/7 via email and chat."}

Save this as data/faq.jsonl. Keep the file under a few MB; larger datasets will exceed memory during training.

4.3 Write a Modelfile#

Create a file named Modelfile in the project root:

cat > Modelfile <<'EOF'
FROM tinyllama

# LoRA parameters – adjust based on your model size
PARAMETER lora_rank 8
PARAMETER lora_alpha 16
PARAMETER lora_dropout 0.05
PARAMETER train_batch_size 4
PARAMETER gradient_accumulation_steps 4
PARAMETER learning_rate 1e-4
PARAMETER num_train_epochs 3

# Tell Ollama where to find the training data
DATA ./data/faq.jsonl
EOF

Explanation of key parameters:

Feel free to tweak these; the values above work for a 1.3B model on a 1 GB RAM VPS.

Step 5: Fine‑Tune the Model#

Now start the training process:

ollama create tinyllama-faq -f Modelfile

Ollama will:

1. Load the base model.
2. Apply LoRA adapters.
3. Iterate over the JSONL data.
4. Save the adapted model as tinyllama-faq.

Monitoring – You can watch progress via logs:

journalctl -u ollama -f

Training on a $5 VPS will take several hours (maybe 4‑8 h depending on dataset size). Expect occasional OOM warnings; if they appear, reduce train_batch_size or lora_rank and restart.

When finished, you’ll see a line like:

Successfully created model tinyllama-faq

Verify:

ollama list
# tinyllama-faq should appear

Step 6: Install OpenWebUI#

OpenWebUI provides a ChatGPT‑style interface that talks to any Ollama model via its API.

6.1 Install Node.js (required for the UI)#

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs

6.2 Grab the OpenWebUI source#

git clone https://github.com/open-webui/open-webui.git
cd open-webui

6.3 Install dependencies and build#

npm ci
npm run build

6.4 Create a systemd service for OpenWebUI#

Create /etc/systemd/system/openwebui.service:

[Unit]
Description=OpenWebUI frontend
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/opt/open-webui
ExecStart=/usr/bin/npm start
Restart=on-failure
Environment=NODE_ENV=production
Environment=OLLAMA_HOST=http://127.0.0.1:11434

[Install]
WantedBy=multi-user.target

Then:

sudo mv open-webui /opt/
sudo chown -R www-data:www-data /opt/open-webui
sudo systemctl daemon-reload
sudo systemctl enable --now openwebui

Check status:

systemctl status openwebui

OpenWebUI runs on port 3000 by default. If your VPS has a firewall, allow the port:

sudo ufw allow 3000/tcp

Step 7: Configure OpenWebUI to Use the Fine‑Tuned Model#

OpenWebUI reads model names from the Ollama API. After the service is running, open your browser to http://<your-vps-ip>:3000.

1. Click the Settings (gear) icon → Model.
2. In the dropdown, you should see tinyllama-faq. Select it.
3. (Optional) Set a custom system prompt under Advanced → Prompt to steer the model’s behavior.
4. Save settings.

Now you can chat with the fine‑tuned model directly from the UI. Try a question from your FAQ dataset:

What is the refund policy?

You should see a response closely matching the training data. If the answer is generic, the model may need more training epochs or a larger LoRA rank.

Step 8: Test, Optimize, and Consider Limitations#

8.1 Basic Performance Check#

time ollama run tinyllama-faq "Explain quantum computing in two sentences."

On a 1 GB VPS, expect 2‑5 seconds per token generation for short answers. Longer outputs will be slower.

8.2 Memory Monitoring#

While the model is loaded, check RAM usage:

free -h
top -p $(pgrep ollama)

If you see swap usage spiking heavily, consider:

• Reducing lora_rank to 4.
• Lowering train_batch_size to 2.
• Using an even smaller base model (e.g., phi-2).

8.3 When to Upgrade#

If you find the latency unacceptable for interactive use, a $10‑$15 VPS with 2‑4 GB RAM will give you a noticeable boost, allowing you to run larger models (e.g., 3B‑7B) with better quality.

Comparison Table: Ollama vs Alternatives on Low‑End VPS#

Ollama wins on simplicity and low‑maintenance for a $5 VPS.

FAQ#

1. Can I fine‑tune a 7B model on a $5 VPS?
No. A 7B model in 4‑bit quantization still needs ~3‑4 GB RAM just to load, leaving little room for training. Stick to ≤1.3B models or use heavier quantization (e.g., GGUF Q2_K) if you must go larger, but expect slower generation.

2. What if my VPS gets killed during training?
Check /var/log/syslog or dmesg for OOM killer messages. Reduce train_batch_size, lora_rank, or gradient_accumulation_steps. You can also train in smaller chunks by splitting your JSONL file and running multiple ollama create commands sequentially, merging adapters later (advanced).

3. Do I need a domain name or SSL for OpenWebUI?
Not strictly. For testing, accessing via IP is fine. For production, put a reverse proxy (nginx or Caddy) in front, obtain a free Let’s Encrypt certificate, and proxy http://localhost:3000. This also lets you run OpenWebUI on port 80/443.

4. How do I update the model after adding more data?
Repeat the fine‑tuning steps: update the JSONL, adjust the Modelfile if needed, and run ollama create with a new model name (e.g., tinyllama-faq-v2). OpenWebUI will pick up the new name automatically after you refresh the model dropdown.

5. Is my data safe? Is anything sent outside the VPS?
All processing stays on your VPS. Ollama and OpenWebUI communicate via localhost HTTP. No data leaves unless you explicitly configure an external proxy or enable telemetry (both are opt‑out by default).

Conclusion#

Fine‑tuning an LLM on a $5 VPS is entirely possible when you match model size to hardware and leverage efficient techniques like LoRA. By installing Ollama, preparing a modest dataset, and serving the result through OpenWebUI, you gain a private, customizable AI assistant without relying on costly cloud GPUs or compromising data privacy.

Remember to keep expectations realistic: the goal is a helpful, domain‑aware chatbot, not a state‑of‑the‑art research model. If you outgrow the tiny VPS, upgrading to a slightly larger plan will let you experiment with bigger parameters and richer datasets—all while retaining the same simple workflow.

For more tutorials on self‑hosted AI, VPS optimization, and developer tooling, keep an eye on mahbuburriad.com.

Happy hacking!