FEMS Project #1 — Comparing Low-Spec Local Setup (Ollama + bge-m3 + Chroma) vs Claude API RAG

Starting a new series — a project log for FEMS (Factory Energy Management System).

I previously worked on running object detection on an edge AI board at my company, and the idea of combining factory / industrial data with AI has stuck with me ever since. As an extension of that, I decided to build a RAG prototype while studying the FEMS domain. “I studied it” doesn’t feel as concrete as “I got this far actually running it.”

Technologies I’ll touch on in this series:

• RAG
• Hugging Face, Ollama
• Embeddings, vector DB
• Token usage strategy / local vs cloud comparison

RAG, embeddings, and vector DBs are somewhat familiar to me already since I covered them recently in Building RAG From Scratch and Building a RAG System with Chroma. Local LLM (Ollama) is the new axis added here.

Table of contents

Spec check — VRAM 6GB is the deciding factor

VRAM is what determines cost-efficiency and speed for local LLMs. With 6GB VRAM, larger models won’t run. Even a 7B model is tight without 4-bit quantization. Anything above 13B is practically impossible.

So I set the direction of the project as a comparative experiment:

In a low-spec on-premise environment, local inference vs calling the Claude / OpenAI API — which is more reasonable, and under what circumstances?

This question also fits well with the FEMS domain itself. Factories tend to have strict rules about data leaving the premises, so there’s a clear demand for on-premise inference. At the same time, introducing GPUs is costly, so how far you can get on low-spec hardware becomes a practical question.

Data — how to gather FEMS domain documents

RAG needs source documents to base its answers on. Candidates in the FEMS domain:

• FEMS manuals / guideline documents
• Energy data analysis reports
• Measurement values from the FEMS DB

For now, I’ll write test guideline documents myself to verify that the RAG pipeline works.

Setup — Python scaffolding + Ollama

Basic skeleton:

• Python project scaffolding
• Installing Ollama (local LLM runtime)
• An adapter that calls 3 providers (Ollama / Claude / OpenAI) through the same interface

Model selection — qwen2.5:7b locally, to fit 6GB VRAM

For the 3-provider comparison, I chose the following models for each:

For the local model, VRAM is what dictates model choice. A 7B model with 4-bit quantization needs roughly 4–5GB VRAM, which fits within 6GB. Anything above 13B isn’t even worth trying. I picked qwen2.5 because it strikes a good balance between Korean language and reasoning performance within the 7B class.

For the cloud side, I deliberately set up top-tier (Opus 4.8) vs mid-tier (gpt-4o) to cover a range of price and quality. This way I can measure “how far the local model keeps up” against two different benchmarks.

Smoke test — first run (cold start)

I first checked whether all three providers answer the same question correctly:

python -m scripts.smoke_test

All three answered correctly. The differences are in latency and cost:

• Opus 4.8 is about 5x more expensive than gpt-4o ($0.00213 vs $0.00041)
• ollama (qwen2.5:7b) took 95 seconds — due to the cold start. Since it was the first call, most of that time was spent loading the model.

Smoke test — second run (after warm-up)

I ran it again with the model already loaded in memory:

• ollama: 95s → 10.00s (result of eliminating the cold start)
• claude: 2.72s → 2.46s
• openai: 6.43s → 2.62s

The gap between 95 seconds and 10 seconds is operationally huge. In other words, how you handle “the first call” is the key to latency for local LLMs:

• Keep the model resident in memory at all times (memory cost)
• Swap it based on usage frequency (cold start each time)

This becomes a core question in designing the operating mode.

Embedding model selection — Ollama’s bge-m3

For embeddings, I settled on Ollama’s bge-m3.

Reasons for choosing it:

• Strong Korean / multilingual performance — since the FEMS guidelines are in Korean, this is a critical requirement
• No separate embedding API needed — Ollama provides both the LLM and embeddings
• Fully local — embeddings are handled without any external API calls

Options I considered but didn’t use:

One thing to note — Chroma’s default embedding function is English-specialized. Feeding it Korean documents directly degrades embedding quality. So I computed the embeddings myself and injected them into Chroma instead.

In my previous post on building RAG from scratch, when I ran Korean text through OpenAI embeddings, the absolute similarity scores came out low, around 0.3 to 0.5. Since bge-m3 specializes in multilingual embeddings, checking whether this improves is one of the key points of this test.

Building the index — starting with 5 chunks

I split the test guideline documents into chunks of 800 characters or less and indexed them:

python -m scripts.build_index

A total of 5 chunks:

• Compressed air system efficiency guideline
• Peak power management guideline
• HVAC optimization guideline
• Energy-saving lighting guideline
• (1 more)

Verifying RAG behavior — all 3 questions correct

I checked whether the sources matched correctly for the given questions:

python -m scripts.rag_demo

All 3 were correct. (Lower cosine distance means more similar — 0 means identical, 1 means unrelated.)

The second question (peak demand), in particular, with a distance of 0.272, matched exceptionally well. This means bge-m3 handles Korean properly.

In my previous post, when I ran Korean text through OpenAI’s text-embedding-3-small, the top similarity scores fell between 0.3 and 0.5. With bge-m3, the distances come out at 0.27–0.39 (lower is better — converting to similarity gives roughly 0.6–0.73). The difference in multilingual embeddings shows up clearly in the numbers.

Retrospective — the backbone is up

What I built out today:

• Python scaffolding ✅
• Ollama running locally ✅
• Simultaneous calls to 3 providers (ollama / claude / openai) ✅
• bge-m3 embeddings + Chroma index ✅
• 5-chunk RAG demo, 100% accuracy ✅

Next, I’ll bring in actual public data PDFs (the Korea Energy Agency’s FEMS implementation guide, energy-saving technical materials, etc.), increase the number of chunks, and move toward comparing local vs cloud answer quality using real domain questions.

Things to study further

1. Ollama operating modes — keep-alive vs swap

• Controlling how long a model stays resident in memory via the OLLAMA_KEEP_ALIVE environment variable
• The cost of swapping when running multiple models on 6GB VRAM
• Automatic swap strategies based on usage frequency

2. bge-m3 vs other Korean embeddings

• KoSimCSE / KoSBERT — the de facto standard for Korean sentence embeddings
• multilingual-e5-large — a multilingual model with decent Korean performance
• Benchmarking embeddings on the same document set (comparing top-1 distance)

3. Why Chroma’s default embedding function is English-specialized

• Chroma’s default is all-MiniLM-L6-v2 (sentence-transformers, English-centric)
• Injecting embeddings directly is the standard approach — set embedding_function=None when creating a collection
• Differences in embedding policy across other vector DBs like pgvector

4. Token usage strategy — from a RAG perspective

• The relationship between the context window and number of chunks × chunk size
• Using prompt caching to cache system prompts / tool definitions
• The impact of the K value in top-K retrieval on cost

5. The FEMS domain itself

• Korea Energy Agency’s FEMS implementation guide (public PDF)
• Energy-saving technical materials — industrial sector (Korea Energy Agency)
• Peak management / compressed air / HVAC / lighting — the four common areas of savings
• Measurement items: power (kW, kWh), power factor, demand, gas/steam, and various other energy flows

6. Trade-offs of adopting on-premise LLMs

• Data security ↑ vs model quality ↓
• Initial GPU cost vs API call cost (break-even point calculation)
• Model update cycle — cloud updates automatically, on-premise requires manual fine-tuning / replacement